This example uses Windows Server 2012 R2. The original root and data volumes are both 100 GB gp3 volumes, and the target size is 50 GB each.

Create an AMI before starting, test the procedure in a non-production environment, and schedule the root-volume migration during a maintenance window. Stopping an instance can lose instance-store data and an automatically assigned public IP address (an Elastic IP is not affected).

AWS documentation: EBS volume modification limitations

1. Migrate the data volume (D: to a new 50 GB volume)

The data volume can be copied online, without downtime.

1. Create and attach the new EBS volume

Create a 50 GB EBS volume in the same Availability Zone as the instance and attach it. In Windows Disk Management, bring it online, initialize it with the same partition style as the source (MBR or GPT), create a simple NTFS volume, and assign drive letter E:.

2. Copy the data with robocopy

robocopy D:\ E:\ /MIR /COPYALL /R:1 /W:1 /LOG:C:\robocopy-d.log

• /MIR: mirror the directory tree.
• /COPYALL: equivalent to /COPY:DATSOU; copies data, attributes, timestamps, NTFS security, owner, and auditing information.
• /R:1 /W:1: retry once and wait one second.
• /LOG: write a log for review.

3. Verify the data volume

Check the log, permissions, and sample files:

type C:\robocopy-d.log | findstr /i "FAILED"
icacls E:\PermTest\*
dir E:\PermTest
type E:\PermTest\admin-only.txt

Confirm that the copy completed without failed files and that representative NTFS permissions match the source volume.

4. Swap the drive letter online

In Disk Management, change the original D: volume to a temporary letter such as X:, then change the new E: volume to D:. Verify that applications can access D: normally.

Get-WmiObject Win32_LogicalDisk -Filter "DeviceID='D:'" | Select Size, FreeSpace

Keep the original volume offline until validation is complete. Then detach it from EC2 and delete it only after confirming that the new volume is healthy.

2. Migrate the system volume (C: to a new 50 GB volume)

System files are locked while Windows is running, so the root volume must be cloned offline from a helper instance.

1. Record the original instance information

Record the original instance Availability Zone, instance ID, root volume ID, root device name, and partition style before making changes.

2. Prepare the helper instance

Launch a Windows Server helper instance in the same Availability Zone and install DiskGenius (the free edition is sufficient): DiskGenius download.

3. Stop the original instance and detach the root volume

Stop the original instance, wait for the Stopped state, and detach its root volume.

4. Create a new 50 GB volume

Create a new 50 GB gp3 volume in the same Availability Zone as the helper and original instances.

5. Attach the original and new volumes to the helper

Attach both the original 100 GB root volume and the empty 50 GB volume to the helper instance. In Disk Management, bring the original volume online but cancel the initialization wizard for the new volume. Do not format or partition the target; DiskGenius will write the cloned disk layout.

6. Clone the disk with DiskGenius

1. Confirm that the helper system disk, the original root disk (including System Reserved and C:), and the empty target disk are visible.
2. Choose Tools → Clone Disk.
3. Select the original root disk as the source and the 50 GB volume as the target.
4. Because the target is smaller, choose Copy files rather than sector-by-sector or filesystem-structure cloning.
5. Adjust the partition boundaries if DiskGenius allocates excessive space to System Reserved. Leave enough space for the Windows partition.
6. Start the task and choose Execute after locking.

7. Rebuild the boot files

After file-level cloning, the disk signature changes and the BCD partition references may be invalid. Windows can then fail with 0xc0000225 winload.exe. Bring the new volume online and assign drive letters in Disk Management.

If the disk has a System Reserved partition (for example F:) and the Windows partition is G:, run:

bcdboot G:\Windows /s F: /f BIOS /l en-us

If there is only one partition (G:), run:

bcdboot G:\Windows /s G: /f BIOS /l en-us

Confirm that the command reports that the boot files were successfully created. Then mark the boot partition active. Replace the disk and partition numbers with the values shown by list disk and list partition:

diskpart
list disk
select disk 2
list partition
select partition 1
active
exit

8. Attach the new volume as the original instance root

Detach both volumes from the helper instance. Attach the 50 GB clone to the original instance as /dev/sda1, the root-device position.

9. Start the original instance and verify

Start the instance. Connect over RDP and confirm that Windows boots normally and that the system disk is the expected size.

3. Final validation

Get-WmiObject Win32_LogicalDisk |
  Select DeviceID,
    @{N='SizeGB';E={[math]::Round($_.Size/1GB,1)}},
    @{N='FreeGB';E={[math]::Round($_.FreeSpace/1GB,1)}}

systeminfo | findstr /i "boot"

Are NTFS permissions preserved?

• Data volume: /COPYALL preserves DACL, SACL, owner, and auditing data. Compare the result with icacls.
• Root volume: DiskGenius copies the NTFS file system, including permission metadata in $MFT and $Secure.

No additional permission migration tool is required. Always verify representative administrator-only and everyone-read files before deleting the original volumes.

References

1. AWS EBS volume modification limitations
2. AWS detaching an EBS volume
3. DiskGenius Clone Disk
4. EC2 Windows EBS volume shrinking in practice (CSDN)

Confirm the situation

cd ~/.hermes/hermes-agent
git log --oneline -1
git describe --tags --always
git remote -v

Compare the latest release with upstream main. If the release is unchanged but main moved forward, automatic updating is behaving as designed.

Workflow

1. Back up configuration and inspect the workspace

Back up configuration and inspect local changes:

cp ~/.hermes/config.yaml ~/.hermes/config.yaml.bak.$(date +%Y%m%d_%H%M%S)
git status --short

Handle local changes first. Generated build output should not be edited or committed manually; commit valuable source changes to your own branch before synchronizing.

2. Fetch upstream

Fetch the official repository, not just your fork:

git fetch upstream main

origin normally points to your fork, while upstream points to the official repository. Running only git pull origin main may leave you on outdated fork code.

3. Merge or rebase

If the fork has no additional commits:

git merge upstream/main --ff-only

If the fork has its own commits, rebase instead:

git rebase upstream/main

Resolve conflicts, stage the files, and run git rebase --continue.

4. Update the virtual environment and verify

Install the editable package and verify it:

./venv/bin/pip3 install -e .
./venv/bin/python3 -m hermes_cli.main --version
./venv/bin/python3 -m hermes_cli.main doctor

Restart Hermes Desktop after the checks pass; an already-running process will not load the new source automatically.

Key conclusions

• No new release means no automatic desktop update; that is expected.
• fetch upstream plus merge/rebase is the core fork-sync workflow.
• User configuration, skills, and sessions are separate from the source checkout, but should still be backed up.
• Extra commits may appear in git describe while the release version remains unchanged.
• Pin a known-good commit and keep a rollback point; git reflog can recover the pre-rebase state.

Architecture

Command request
  ↓
Approval policy
  ↓
Sandbox type
  ↓
Seatbelt / Bubblewrap + seccomp / Restricted Token

Approval answers whether an action needs confirmation; sandboxing limits what an approved process can touch.

macOS Seatbelt

The macOS backend loads an SBPL profile through sandbox-exec. A profile starts from denied capabilities and adds exceptions for the workspace, temporary directories, interpreters, and network policy.

• read-only does not allow workspace writes.
• workspace-write opens the workspace and explicit writable_roots.
• Network access is a separate policy; write access does not imply network access.
• Child processes generally inherit the process restriction.

Linux backends

Bubblewrap builds the process filesystem view with read-only system mounts and writable workspace bindings. seccomp-BPF filters system calls; it is not a path permission system. Landlock provides path-oriented kernel access control and can restrict opening, writing, executing, and deleting files, subject to kernel support.

no_new_privs prevents setuid and file capabilities from granting additional privilege and is an important companion to seccomp and Landlock.

Windows Restricted Token

The Windows backend uses a Restricted Token together with system ACLs and child-process rules. It does not map one-to-one to macOS profiles or Linux namespaces, so it should not be understood only as a directory read/write model.

How it differs from Docker

Docker Desktop on macOS normally runs Linux containers inside a Linux VM provided by Apple Virtualization.framework. Container isolation happens inside that VM and is a different layer from Seatbelt restrictions applied to a host process.

| Mechanism | Main boundary | Typical capability |
|

One-shot benchmarks

NodeQuality

NodeQuality is the simplest all-in-one option. It covers CPU, disk, network, IP quality, streaming access, and route information:

bash <(curl -fsSL https://raw.githubusercontent.com/xykt/NodeQuality/main/NodeQuality.sh)

YABS, bench.sh, and SuperBench

curl -fsL yabs.sh | bash
wget -qO- bench.sh | bash
bash <(wget -qO- https://raw.githubusercontent.com/spiritLHLS/superbench/main/superbench.sh)

These scripts run several tests at once and may write heavily to disk. Avoid business hours on production hosts.

Component tests

CPU

# Run the official Linux CLI after downloading Geekbench
./geekbench6

# UnixBench
git clone https://github.com/kdlucas/byte-unixbench.git
cd byte-unixbench/UnixBench && make && ./Run

Disk

fio is better suited than dd for comparing IOPS and queue depth. Do not run long random-write tests directly against a production disk.

sudo apt install -y fio
fio --name=randread --filename=/tmp/fio.test --size=1G --bs=4k \
  --rw=randread --ioengine=libaio --direct=1 --runtime=60 --time_based

Network, streaming, and IP quality

speedtest
bash <(curl -fsSL https://raw.githubusercontent.com/i-abc/speed-test/main/speedtest.sh)
iperf3 -c <server_ip> -t 30
bash <(curl -fsSL https://raw.githubusercontent.com/lmc999/RegionRestrictionCheck/main/check.sh)
bash <(curl -fsSL https://raw.githubusercontent.com/xykt/IPQuality/main/ip.sh)

Return routes

curl -fsSL https://raw.githubusercontent.com/zhanghanyun/backtrace/main/install.sh | sh
curl nxtrace.org/nt/install.sh | bash
nexttrace <target_ip>

Recommended workflow

Start with NodeQuality, then repeat with YABS, IPQuality, a streaming check, and backtrace when needed. Record the instance type, region, test time, disk type, and network limits so results remain comparable.

Safety notes

• Read remote scripts before running them; do not blindly use curl | bash on production hosts.
• Disk tests can create substantial writes. Use a temporary file, never a business data path.
• Public iperf3 servers are variable; treat their results as approximate.
• Do not publish public IPs, complete routes, or secrets with benchmark logs.
• Old bash.icu, ilemonra.in, and check.unlock.moe links are no longer reliable; use project repositories.

| Target | Recommended tools |
|

Layers

| Layer | Method | Contents |
|

Background

| Item | Info |
|

Symptom

The user sent a message to the Hermes bot via Telegram:

• The bot showed "typing..." but never replied
• After several minutes, a No response from provider for 180s error arrived
• At the same time, the terminal CLI using the same model worked perfectly, 5-10s per response

Error log:

⚠️ No response from provider for 180s (model: mimo-v2.5-pro, context: ~20,614 tokens). Reconnecting...
⚠️ API call failed (attempt 1/3): APITimeoutError
   🔌 Provider: custom  Model: mimo-v2.5-pro
   🌐 Endpoint: https://YOUR-API-ENDPOINT/v1
   📝 Error: Request timed out.

Investigation

Initial hypothesis (wrong direction)

At first I suspected the API was unstable or the streaming timeout was too short: the log showed Stream stale for 180s (threshold 180s) — no chunks received, suggesting raising HERMES_STREAM_STALE_TIMEOUT=300. But the user pointed out the CLI with the same model worked perfectly.

Check the proxy config

The system environment had a proxy:

$ env | grep -i proxy
HTTP_PROXY=http://127.0.0.1:10808
HTTPS_PROXY=http://127.0.0.1:10808
ALL_PROXY=socks5h://127.0.0.1:10808

The proxy is Xray on port 10808. Testing the API through it worked:

$ curl --proxy http://127.0.0.1:10808 https://YOUR-API-ENDPOINT/v1/models
# 200, 0.8s — fine

The proxy itself was fine.

Telegram session stuck in a loop

The agent.log showed the Telegram session stuck in a loop: receive message, call the API, get killed after 180s of no data, gateway restarts and resumes the session, stuck for 180s again... Deleting the stuck session and restarting the gateway didn't help; new sessions got stuck the same way.

Comparing CLI vs Telegram API calls

Key observation: the CLI session's API call completed in 5-10s; the Telegram session's API call timed out at 180s. Same endpoint, same model, completely different behavior.

Gateway blocked

The gateway.log stopped producing output after a certain point: the cron ticker stopped printing, new messages stopped being recorded, but the CLI session's agent.log kept writing normally. This meant the gateway process's asyncio event loop was fully blocked by the stuck HTTP request.

The launchd environment difference (root cause)

Checking the launchd plist:

<key>EnvironmentVariables</key>
<dict>
    <key>PATH</key>
    <string>~/.hermes/hermes-agent/venv/bin:...</string>
    <key>VIRTUAL_ENV</key>
    <string>~/.hermes/hermes-agent/venv</string>
    <key>HERMES_HOME</key>
    <string>~/.hermes</string>
</dict>

No HTTP_PROXY, HTTPS_PROXY, or ALL_PROXY!

And the .env:

# HTTP_PROXY=http://127.0.0.1:10808      ← commented out!
# HTTPS_PROXY=http://127.0.0.1:10808     ← commented out!
# ALL_PROXY=socks5h://127.0.0.1:10808    ← commented out!

Case closed:

| Process | Started by | Proxy | API behavior |
|

Background

| Item | Info |
|

Test Environment

| Item | Configuration |
|

Test Environment

| Item | Configuration |
|

Test Environment

| Item | Configuration |
|

Test Environment

| Item | Configuration |
|

Architecture

The NAT-GW instance has two ENIs:

• ENI1 (Public): public subnet, has an IGW route, with an EIP attached
• ENI2 (Private): private subnet, receives client egress traffic

Traffic path: Client → VPC route table → ENI2 (Private) → RRAS NAT (SNAT to EIP) → ENI1 (Public) → IGW → Internet

Prerequisites

• Both ENIs of the NAT-GW have Source/Dest Check disabled
• The private subnet route table has 0.0.0.0/0 → NAT-GW ENI2
• Security groups allow traffic from the client subnet to the NAT-GW

Steps

1. Create a dual-ENI instance

# Create instance in public subnet with auto-assigned public IP
aws ec2 run-instances --image-id ami-0668ff427a68f0066 `
  --instance-type t3.medium `
  --subnet-id subnet-public `
  --iam-instance-profile Name=EC2SSM `
  --associate-public-ip-address `
  --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=NAT-GW}]"

# Disable source/dest check
aws ec2 modify-instance-attribute --instance-id i-xxxxxxxxx --no-source-dest-check

# Create the second ENI (private subnet)
aws ec2 create-network-interface --subnet-id subnet-private `
  --groups sg-xxxxxx --description "NAT-GW internal ENI"

# Attach to instance
aws ec2 attach-network-interface --network-interface-id eni-xxxxxxxxx `
  --instance-id i-xxxxxxxxx --device-index 1

# Disable source/dest check on the second ENI too
aws ec2 modify-network-interface-attribute `
  --network-interface-id eni-xxxxxxxxx --no-source-dest-check

2. Configure the private subnet route table

aws ec2 create-route --route-table-id rtb-xxxxxxxxx `
  --destination-cidr-block 0.0.0.0/0 `
  --network-interface-id eni-xxxxxxxxx

3. Install the RRAS role

Administrator PowerShell:

Install-WindowsFeature -Name Routing -IncludeManagementTools
Install-RemoteAccess -VpnType RoutingOnly
Get-Service RemoteAccess
# Status: Running

4. Rename adapters and remove the internal default route

Rename-NetAdapter -Name 'Ethernet 3' -NewName 'Public'
Rename-NetAdapter -Name 'Ethernet 4' -NewName 'Private'

# Remove the default route on the Private adapter
Remove-NetRoute -InterfaceAlias 'Private' -DestinationPrefix '0.0.0.0/0' -Confirm:$false

# Prevent DHCP lease renewal from restoring the default gateway
$adapter = Get-NetAdapter -Name 'Private'
$guid = $adapter.InterfaceGuid
$regPath = "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\$guid"
Set-ItemProperty -Path $regPath -Name 'DhcpDefaultGateway' -Value @('') -Type MultiString

Without this registry change, DHCP renews the lease hourly and re-adds the default route, which conflicts with the Public adapter's default route and causes intermittent egress packet loss.

5. Stop the ICS service

netsh routing ip nat install refuses to install if the SharedAccess service is running. Just stop ICS; leave the firewall on:

Stop-Service SharedAccess -Force -ErrorAction SilentlyContinue
Set-Service SharedAccess -StartupType Disabled

Once NAT is configured, having all three Windows Firewall profiles enabled does not affect NAT forwarding — RRAS NAT runs in the kernel via ipnat.sys and bypasses the WFP rule chain.

6. Enable IP forwarding

Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters' `
  -Name IPEnableRouter -Value 1

Set-NetIPInterface -InterfaceAlias 'Public' -Forwarding Enabled
Set-NetIPInterface -InterfaceAlias 'Private' -Forwarding Enabled

Restart-Service RemoteAccess -Force
Start-Sleep -Seconds 5

ipconfig /all | Select-String 'IP Routing'
# IP Routing Enabled. . . . . . . . : Yes

7. Configure RRAS NAT

netsh routing ip nat install

# Public interface as NAT egress (full = address and port translation)
netsh routing ip nat add interface name="Public" mode=full

# Private interface as internal (private = forward only)
netsh routing ip nat add interface name="Private" mode=private

netsh routing ip nat show interface

8. Verify

On the client instance:

curl.exe -s http://checkip.amazonaws.com
# Egress IP should be the NAT-GW EIP

ping 1.1.1.1

Pitfalls

Dual default route (most common)

The second ENI gets a default route via DHCP with the same metric (20) as the primary adapter. Windows does ECMP load balancing, so half the egress traffic goes out the Private adapter — which has no IGW route and drops the packets. Symptom: intermittent ping/curl failures, occasional TCP timeouts.

Fix: remove the Private adapter's default route + clear DhcpDefaultGateway in the registry, or set the Private adapter InterfaceMetric very high (e.g., 1000).

New-NetNat does not work on physical adapters

New-NetNat only works with Hyper-V Internal Virtual Switch. On EC2 adapters it shows Active=True but performs no SNAT on forwarded traffic.

ICS hijacks the IP address

When ICS is enabled, it automatically changes the Private interface IP to 192.168.137.1, overriding the DHCP-assigned VPC address. Packets with mismatched IPs are dropped.

RRAS does not recognize a hot-attached ENI

After attaching the second ENI, you must run Restart-Service RemoteAccess, otherwise netsh routing ip show interface will not see the new adapter.

References

• NAT instances - Amazon VPC User Guide
• Install-RemoteAccess
• RAS Gateway

Symptom

Three Windows instances failed sysprep. The log at C:\Windows\System32\Sysprep\Panther\setuperr.log shows:

Error  SYSPRP Package NotepadPlusPlus_1.0.0.0_neutral__2247w0b46hfww was installed for a user, but not provisioned for all users.
Error  SYSPRP Failed to remove apps for the current user: 0x80073cf2.

Even after uninstalling the Notepad++ desktop version, sysprep still failed.

Root Cause

Notepad++ 8.9 and later register an AppX package (NotepadPlusPlus_1.0.0.0_neutral__2247w0b46hfww). Uninstalling only the desktop version does not remove the AppX package, so sysprep cannot remove the user app.

This is a known Notepad++ issue (GitHub Issue #13775).

Resolution

1. Remove the AppX package

# Remove the Notepad++ AppX package for all users
Get-AppxPackage -AllUsers *NotepadPlusPlus* | ForEach-Object {
    Remove-AppxPackage -Package $_.PackageFullName -AllUsers -ErrorAction SilentlyContinue
}

# Remove the provisioned AppX package
Get-AppxProvisionedPackage -Online | Where-Object {$_.DisplayName -like "*NotepadPlusPlus*"} | 
    Remove-AppxProvisionedPackage -Online

2. Run Sysprep

Run sysprep with EC2Launch v2 using default settings. After removing the AppX package, sysprep succeeds.

& "C:\Program Files\Amazon\EC2Launch\EC2Launch.exe" sysprep

Summary

• Uninstalling only the Notepad++ desktop version does not fix the issue; the AppX package must be removed
• Avoid installing Notepad++ 8.9+ before creating a custom AMI, or validate sysprep compatibility beforehand
• Create an AMI or snapshot backup before running sysprep

References

• Troubleshoot Amazon EC2 Windows instance Sysprep issues
• Run Windows Sysprep using EC2Launch v2

Symptom

After boot, the system time is 8 hours behind. After a while (depending on the NTP poll interval), W32Time synchronizes with the NTP source and applies a one-time 8-hour jump, causing application log timestamp chaos, Kerberos authentication failures, and scheduled task anomalies.

This is common on Windows instances with the timezone set to UTC+8 (China Standard Time), especially after stop/start, reboot, or restore from AMI.

Error Message

System event log (Event ID 1, source Microsoft-Windows-Kernel-General):

System time changed from 2026-06-16T01:52:47Z
to 2026-06-16T09:52:47Z
Delta: 28800135 ms (~8 hours)

RTC time in UTC: false

The key field: RTC time in UTC: false means the hardware RTC clock stores local time, not UTC.

Root Cause

• Windows assumes by default that the RTC hardware clock stores local time
• The EC2 Nitro platform RTC stores UTC time
• When RealTimeIsUniversal is not set, Windows interprets the UTC value in the RTC as local time on boot, then subtracts the timezone offset to get UTC — resulting in 8 hours behind
• W32Time then detects the offset and applies a one-time correction (+8h jump)

Solutions

Create an AMI snapshot backup before proceeding.

Option 1: Set RealTimeIsUniversal (recommended)

Tell Windows to treat the RTC hardware clock as UTC:

reg add "HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /t REG_DWORD /d 1 /f

Option 2: Disable VMICTimeProvider

EC2 instances do not need the Hyper-V time sync provider; disable it to avoid interference:

reg add "HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider" /v Enabled /t REG_DWORD /d 0 /f
Restart-Service w32time

Option 3: Use Amazon Time Sync Service

Switch the NTP source to the EC2 link-local time service (169.254.169.123), which has very low latency and does not require internet access:

w32tm /config /manualpeerlist:"169.254.169.123,0x9" /syncfromflags:manual /reliable:no /update
Restart-Service w32time
w32tm /resync /force

Shorten the poll interval:

reg add "HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpClient" /v SpecialPollInterval /t REG_DWORD /d 900 /f
Restart-Service w32time

Combined recommendation

All three options are compatible and can be applied together. Apply all, reboot, then verify:

reg query "HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal
w32tm /query /status
w32tm /query /peers

References

• AWS - Set the time for a Windows instance
• AWS - Amazon Time Sync Service
• Microsoft Learn - Windows Time Service registry entries

Background

Clients cannot access the file server \\fileserver01. Security groups or routing (via a firewall ENI) are initially suspected; network connectivity needs to be confirmed.

Environment (from packet capture):

• Client: 10.0.1.100
• FSx default DNS name: amznfsxexample.corp.example.com -> 10.0.2.10
• DNS alias (the name clients actually use): fileserver01.corp.example.com
• Domain controller / KDC: 10.0.3.5

Investigation

Packet capture analysis on the client:

DNS resolution works — the alias is a CNAME ultimately pointing to the FSx:

fileserver01.corp.example.com  CNAME  amznfsxexample.corp.example.com  A  10.0.2.10

Network layer is fine — TCP handshakes to FSx 445 (SMB) and to KDC 88 (Kerberos) both succeed, and SMB2 Negotiate returns success:

10.0.1.100 -> 10.0.2.10:445  SYN/SYN-ACK/ACK normal
SMB2 Negotiate Protocol Response   NT Status: STATUS_SUCCESS

The real failure is in Kerberos — the client requests a service ticket for the alias from the KDC, which returns error code 7:

TGS-REQ  SName: cifs/fileserver01 , cifs/fileserver01.corp.example.com
KRB-ERROR  error_code 7  = KDC_ERR_S_PRINCIPAL_UNKNOWN

Without a ticket, SMB2 Session Setup fails (STATUS_NO_LOGON_SERVERS); the client RSTs and retries repeatedly.

Root Cause

fileserver01 is a DNS alias configured for the FSx file system. When a client accesses a share via the alias, it requests a cifs/fileserver01 service ticket from the KDC. However, the corresponding HOST/ SPN was never registered on the FSx file system's AD computer object, so the KDC cannot find this service principal.

So this is not a security group, route table, or firewall ENI issue — the network is reachable. A quick way to verify: accessing the FSx via its default DNS name amznfsxexample.corp.example.com usually works; if the default name works but the alias does not, it is almost certainly an SPN problem.

When Kerberos is enabled, each DNS alias requires two SPNs:

HOST/fileserver01
HOST/fileserver01.corp.example.com

Resolution

Run on a Windows instance joined to the same AD domain, as administrator. This modifies AD computer object SPNs; perform within a change window.

1. Install the AD PowerShell module

Install-WindowsFeature RSAT-AD-PowerShell

2. Check whether the alias SPN is already taken

SPNs must be unique in the domain. Before adding, confirm the alias HOST/ SPN is not already registered on another computer object (e.g., the old file server before migration):

$ALIAS = "fileserver01.corp.example.com"
SetSPN /Q ("HOST/" + $ALIAS)
SetSPN /Q ("HOST/" + $ALIAS.Split(".")[0])

If already present, remove it from the old object first.

3. Set the alias SPN on the FSx computer object

$FSxDnsName = "amznfsxexample.corp.example.com"
$Alias = "fileserver01.corp.example.com"
$FileSystemHost = (Resolve-DnsName $FSxDnsName | Where Type -eq 'A')[0].Name.Split(".")[0]
$FSxAdComputer = (Get-AdComputer -Identity $FileSystemHost)
Set-AdComputer -Identity $FSxAdComputer -Add @{"msDS-AdditionalDnsHostname" = @($Alias, $Alias.Split(".")[0])}

4. Verify the SPN

$FileSystemDnsName = "amznfsxexample.corp.example.com"
$FileSystemHost = (Resolve-DnsName ${FileSystemDnsName} | Where Type -eq 'A')[0].Name.Split(".")[0]
$FSxAdComputer = (Get-AdComputer -Identity ${FileSystemHost})
SetSpn /L ${FSxAdComputer}.Name

Confirm the output contains HOST/fileserver01 and HOST/fileserver01.corp.example.com.

5. Purge client tickets and retry

klist purge

Re-access \\fileserver01 and confirm the share opens normally.

Summary

This issue had nothing to do with security groups, route tables, or the firewall ENI — the network path was fully reachable. The root cause was the DNS alias missing a HOST/ SPN on the FSx AD computer object, causing the KDC to return KDC_ERR_S_PRINCIPAL_UNKNOWN and SMB authentication to fail. Adding the two SPNs restores access.

Make "configure SPN" a standard onboarding step whenever associating a new DNS alias with an FSx file system.

References

• Accessing data using DNS aliases (FSx for Windows)
• Configure SPNs for Kerberos

Core Concepts

Conversion Eligibility

| Instance Source | Can Convert to BYOL |
|

Emulator Components

Windows 11 ARM ships with multiple emulators (all running in user mode):

| Component | Location | Function |
|

Symptom

After Windows Server 2025 installs the latest cumulative update, SMB access between instances cloned from the same AMI fails authentication with an "incorrect password" error, even though the password is correct.

The system log records Event ID 6167 (LsaSrv):

There is a partial mismatch in the machine ID. This indicates that the ticket
issued by user S-1-5-21-xxx has either been manipulated or it belongs to a
different boot session. Failing authentication.

NOTE: This can also happen if the machine was cloned without using sysprep.

Prerequisites

• Windows Server 2025 with the latest cumulative update installed
• Instance created via AMI clone without running Sysprep

Confirmation

Run on both machines:

$sid = (New-Object System.Security.Principal.NTAccount('Administrator')).Translate([System.Security.Principal.SecurityIdentifier]).Value
$machineSid = $sid.Substring(0, $sid.LastIndexOf('-'))
Write-Output "Machine SID: $machineSid"

If the Machine SID is identical on both, the problem is confirmed.

Temporary Workaround

Disable SID uniqueness check via the registry (Known Issue Rollback mechanism). Both SMB server and client need it:

New-Item -Path 'HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides' -Force
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides' -Name '1517186191' -Value 0 -Type DWord
Restart-Computer -Force

Limitation: Valid until end of 2027; Microsoft will remove this bypass in a future security update.

Permanent Solutions

Option 1: Sysprep to regenerate SID (recommended)

For N machines with the same SID, keep 1 untouched and run Sysprep on the other N-1:

# In EC2, use EC2Launch v2
& "C:\Program Files\Amazon\EC2Launch\EC2Launch.exe" sysprep

Sysprep impact:

| Item | Affected |
|

Prerequisites

• An Amazon Linux 2023 rescue instance in the same AZ as the target EBS volume
• The target Windows instance is stopped
• The target root volume is detached from the original instance and attached to the rescue instance

1. Mount the target volume

# AL2023 kernel ships with the ntfs3 driver
sudo modprobe ntfs3

# Identify the device (use the SERIAL field)
sudo lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT,FSTYPE

# Mount
sudo mkdir -p /mnt/win
sudo mount -t ntfs3 -o rw /dev/nvme1n1p1 /mnt/win

Registry hive files are at /mnt/win/Windows/System32/config/:

| File | Registry hive | Typical use |
|

Configuration Location

Cloudflare Dashboard path:

Domain -> Caching -> Cache Rules

Design Strategy

The goal is not to cache the entire site HTML. Instead, prioritize caching long-lived static resources like CSS, JS, images, and favicon. VuePress build assets typically include content hashes, making them ideal for long Edge TTLs. HTML pages are left uncached to avoid stale content after updates.

Final Rule

Match expression:

(http.host eq "blog.example.com" and
 (starts_with(http.request.uri.path, "/assets/") or
  ends_with(http.request.uri.path, ".css") or
  ends_with(http.request.uri.path, ".js") or
  ends_with(http.request.uri.path, ".mjs") or
  ends_with(http.request.uri.path, ".png") or
  ends_with(http.request.uri.path, ".jpg") or
  ends_with(http.request.uri.path, ".jpeg") or
  ends_with(http.request.uri.path, ".gif") or
  ends_with(http.request.uri.path, ".webp") or
  ends_with(http.request.uri.path, ".svg") or
  ends_with(http.request.uri.path, ".ico") or
  ends_with(http.request.uri.path, ".woff") or
  ends_with(http.request.uri.path, ".woff2") or
  ends_with(http.request.uri.path, ".ttf") or
  ends_with(http.request.uri.path, ".otf")))

Rule action:

Eligible for cache
Edge TTL: Ignore cache-control header and use this TTL
TTL: 1 month

Browser TTL is not overridden separately to avoid prolonged browser caching that could complicate debugging after updates.

Issue: matches Regex Not Available

Initially used a regex expression:

http.request.uri.path matches "^/(assets/|.*\\.(css|js|mjs|png|jpg|jpeg|gif|webp|svg|ico|woff2?|ttf|otf)$)"

Deployment failed with:

not entitled: the use of operator Matches is not allowed,
a Business plan or a WAF Advanced plan is required

The Cloudflare Free plan doesn't support the matches regex operator. The fix was to decompose the regex into a combination of starts_with and ends_with functions, which are available on all plans.

Verification

Check cache hit via response headers:

curl -I https://blog.example.com/favicon.ico

First request:

cf-cache-status: REVALIDATED

Second request:

cf-cache-status: HIT
age: 9

HIT means the request was served from a Cloudflare edge cache. age is the number of seconds the cached object has lived on the edge node.

Monitoring Cache Hit Rate

View overall metrics in the Cloudflare Dashboard:

Domain -> Caching -> Overview

Key metrics:

• Cache Analytics
• Cache hit ratio
• Requests by cache status
• Bandwidth saved

Filter by hostname for blog-specific traffic:

Domain -> Analytics -> HTTP Traffic -> Filter by hostname

Future Considerations

The current rule can be kept long-term. If HTML caching is desired later, consider:

• Refresh delay after homepage/article updates
• Automatic Cloudflare Cache Purge after GitHub Actions deployment
• Shorter TTL for text/html responses
• Priority ordering between Cache Rules and Page Rules

For now, static-only caching is the safer approach.

Project Info

| Item | Value |
|

The stack is VuePress 2 rc.30 + vuepress-theme-hope rc.107 + @vuepress/plugin-slimsearch@2.0.0-rc.130 + slimsearch 2.3.0, bilingual (zh-CN at /, en-US at /en/).

1. customFields formatter crash (split is not a function)

Symptom

After enabling slimsearch with customFields (categories/tags), searching threw this in the console:

n[f.value].split is not a function

No results came back at all.

Root cause

@vuepress/plugin-slimsearch merges the customFields formatter with the default locale on the node side via getFullLocaleConfig / deepAssign. When you write it as an object:

formatter: {
  "/": "Category: $content",
  "/en/": "Category: $content",
}

deepAssign treats this string as an enumerable object and iterates it, so each character index becomes a key and the value becomes a single character. "Category: $content" ends up mangled into {0:"C", 1:"a", ...}. The client then calls .split("$content") on it and throws split is not a function.

This is not a config mistake; it's a bug in how the plugin + @vuepress/helper locale merge handles string values.

Fix

In theme.ts, replace search: true with an explicit slimsearch config and write the formatter as an explicit string object (to bypass the mangled merge path):

slimsearch: {
  indexContent: true, // changed to false later, see section 3
  customFields: [
    {
      getter: (page) => page.frontmatter.category,
      formatter: {
        "/": "Category: $content",
        "/en/": "Category: $content",
      },
    },
    {
      getter: (page) => page.frontmatter.tag,
      formatter: {
        "/": "Tag: $content",
        "/en/": "Tag: $content",
      },
    },
  ],
},

The formatter is a Record<string,string> (locale -> string); $content is replaced with the getter's return value. The client SearchResult component picks the string for the current routeLocale, then split("$content") to stitch the prefix and suffix.

Verification

Searching certificate / DevOps on the English site works, the customField shows "Tag: ..." / "Category: ...", no console error.

2. Too many search results

Symptom

A single query returned a flood: the -> 31 pages / 438 lines, certificate -> 3 pages / 35 lines.

Root cause

indexContent: true indexes the entire body. The slimsearch SearchResult component lists all matching paragraphs for each hit page, and the render loop n.map(...) in the source has no cap. So every paragraph containing the word gets rendered.

theme-hope passes through SlimSearchPluginOptions directly and does not expose combineWith / per-page hit cap / prefix / fuzzy, the native slimsearch options (SlimSearchPluginOptions only has indexContent/sortStrategy/indexOptions(tokenize/processTerm)/customFields/filter, etc.). There's no middle ground of "searchable body but few results".

Fix

slimsearch: {
  indexContent: false, // only index title, headings, excerpt, categories/tags
  ...
}

Measured comparison (English /en/)

| Query | indexContent:true | indexContent:false |
|

Background

Yuque is Alibaba's online knowledge-management tool, with notes stored in the cloud. My Yuque workspace held 27 private knowledge bases and 360+ documents, all of which needed to be exported as local Markdown.

Comparing options

| Option | Tool | Best for | Pros | Cons |
|

The Mac had an IPv6 address but couldn't ping any public IPv6 address. It had always worked, then one day it just stopped, and rebooting the router didn't help. It turned out to be iStoreOS's unstable NDP Relay mode — flipping one line of config from relay to server fixed it. This post walks through the full layer-by-layer investigation down to the root cause.

Environment

| Item | Value |
|

Investigation

1. Confirm RustDesk Process Status

pgrep -x RustDesk
# Outputs PID — RustDesk is running

Process is healthy, ruling out crashes or startup failures.

2. Locate Configuration Files

RustDesk config directory on macOS:

~/Library/Preferences/com.carriez.RustDesk/

Key files:

| File | Purpose |
|

Why the Old Approach Failed

Two separate LaunchAgents existed, each managing different mounts, with these problems:

1. RunAtLoad only fires once — doesn't re-trigger after sleep wake
2. No network state detection — can't sense network recovery
3. No retry mechanism — runs before network is ready, fails silently

New Architecture

┌─────────────────────────────────────────────────┐
│  com.checo.smb-auto (LaunchAgent)               │
│  KeepAlive=true → auto-restart on crash         │
│  RunAtLoad=true → start on login                │
└──────────────┬──────────────────────────────────┘
               │
    ┌──────────▼──────────┐
    │  smb-watcher.sh     │  Background daemon
    │  (always running)   │
    │                     │
    │  Gateway Ping       │  Check every 2 min
    │  offline→online     │  Trigger --force remount
    │                     │
    │  Mount Integrity    │  Missing→auto-remount
    └──────────┬──────────┘
               │ calls
    ┌──────────▼──────────┐
    │  smb-mount.sh       │  Unified mount script
    │  (supports --force) │
    └─────────────────────┘

Implementation Details

1. Unified Mount Script ~/bin/smb-mount.sh

#!/bin/bash
# Unified SMB auto-mount script
# Usage: smb-mount.sh [--force]

FORCE=0
[[ "$1" == "--force" ]] && FORCE=1

# Share definition table: IP|user|password|share|display_name
SHARES=(
    "192.168.x.x|administrator|PASSWORD|C|Win-C"
    "192.168.x.x|administrator|PASSWORD|D|Win-D"
    "192.168.x.x|administrator|PASSWORD|E|Win-E"
    "192.168.x.x|NAS_USER|NAS_PASS|share|NAS"
)

Mount detection key function:

is_mounted() {
    local ip="$1" share="$2"
    # macOS mount output: //user@IP/share on /Volumes/...
    # ⚠️ Must use //.*@IP pattern due to username prefix
    mount | grep "smbfs" | grep -q "//.*@$ip/$share "
}

Gotcha: macOS mount output for SMB includes a username prefix: //administrator@192.168.x.x/C on /Volumes/.... Using //192.168.x.x/C as a pattern will always return false.

Mount method — osascript (Finder way):

osascript -e "tell application \"Finder\" to mount volume \"smb://user:pass@IP/share\""

Why not mount -t smbfs? mount_smbfs requires root. Sudo-mounted volumes belong to root, causing Permission denied for regular users. osascript mounts via Finder with correct permissions.

Finder mount path naming convention:

| Mount Order | Path |
|

Crash Symptom

The game crashed immediately on launch. Crash report:

Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Code Type: X86-64 (Translated)
Triggered by Thread: 8 Main

The crash occurred during main-thread initialization, with Lua-related strings appearing in registers.

Environment Check

| Item | Result |
|

• Information Security graduate
• Geography enthusiast
• Photography lover
• Fan of 房东的猫 (Fangdong's Cat)

For me, blogging isn't about packaging everything neatly — it's about preserving "how I judged the situation at the time, what pitfalls I hit, and how it was ultimately resolved." Next time I encounter a similar problem, I can take fewer detours; when others find it through search, they can quickly judge whether this approach suits them.

AI

The AI category mainly records experiences with large language models and related tools.

Currently planned topics:

• Local LLM deployment, e.g., LM Studio, Ollama.
• Tool chains like DeepSeek, Cherry Studio, SiliconFlow.
• API calls, model integration, and some low-cost usage methods.
• Practical uses of AI tools in daily work.

Articles in this section will be practice-oriented, not just theory. It's only worth recording if it can be run locally, integrated with a client, and solve real problems.

AWS

AWS currently has the most content on the blog. This section mainly covers cloud service troubleshooting, EC2 operations, Windows/Linux system issues, S3, ALB, IAM, FSx, SSM, CloudWatch, and similar cases.

These articles typically follow this structure:

1. Problem symptoms
2. Key logs
3. Investigation path
4. Root cause diagnosis
5. Solution
6. Follow-up recommendations

Many AWS problems look like cloud platform issues, but the root cause may end up being in the operating system, certificate chain, KMS permissions, Windows Update, AD ports, Kerberos tickets, or third-party security software. I write these down as a reminder: troubleshoot in layers, don't attribute the problem to a specific component right from the start.

DevOps

The DevOps category covers more general server, container, and service migration records.

For example:

• 1Panel server maintenance
• Docker networking and iptables/nftables issues
• Migrating new-api from SQLite to MySQL
• Blue-green production service switching
• Reverse proxy and database connection issues

The emphasis in this type of content is reproducibility and rollback capability. Operations in production environments can't just aim for "it works" — you also need to consider backups, verification, traffic switching, and failure rollback.

VPS

The VPS category records servers, networking, domains, and common scripts.

Planned additions:

• Common Linux scripts
• VPS benchmarking and network testing
• Domain, DNS, and certificate configuration
• Nezha Probe, triple-network IPs, proxy and network connectivity

This section will be relatively miscellaneous, but all centered around personal server and network asset management.

macOS

The macOS category mainly records my own Mac usage, troubleshooting, and hardware inspection.

For example:

• Hardware inspection after receiving the Mac mini M4
• VS Code auto-update failure on external APFS volume
• External drives, app migration, system log investigation

These articles lean toward personal experience, but many pitfalls are common. Especially on macOS, issues with cross-volume operations, permissions, temporary directories, and app auto-updates are usually not obvious — until something goes wrong, and then they're hard to pinpoint.

Windows and Security

The Windows category currently covers WSL, Kali, and Windows environment-related content.

Planned additions:

• WSL migration and tinkering
• Kali tool environment
• Windows Server operations experience
• Security testing environment setup

This differs from the Windows troubleshooting in the AWS category: the AWS category focuses more on cloud cases, while the Windows category is more about local environments and personal use.

Photography

Photography is another long-term section.

This section won't focus much on gear specs — it's more about the photos themselves and shooting scenes, like the Southern Anhui Sichuan-Tibet Highway, birds, and football matches. Tech blogs tend to get drier over time; photography keeps this site from being just a problem list and preserves a bit of life.

Writing Principles

I'll try to follow several principles for future articles:

• Don't include sensitive information — accounts, instance IDs, internal IPs, domains, and keys should all be generalized.
• Write less vague conclusions, more reasoning behind decisions.
• Commands should be directly copyable, but risk warnings must be included.
• Explain root causes, not just steps.
• Production operations must include backup and rollback reminders.
• Unfinished drafts shouldn't be pinned; placeholder content shouldn't be treated as published articles.

Future Plans

Short-term: clean up existing notes, especially AWS, DevOps, and VPS-related content. Medium-term: further organize the blog's visual design and navigation to make categories clearer and article lists more browsable.

This blog isn't very extensive yet, but it's already developing its own direction: technical problems don't just record "how to do it" but more importantly "why it was done this way." With long-term accumulation, it will become increasingly valuable.

1. The DeepSeek official website has been extremely popular recently and often fails to respond
   

2. Local deployment offers higher security

3. Local deployment can bypass some official restrictions

Test Environment

OS: Windows 10 Pro 22H2
CPU: AMD Ryzen 5 5600H (6C12T, Base 3.3GHz / Boost 4.2GHz)
GPU: NVIDIA GeForce RTX 3050 Ti Laptop GPU (4GB GDDR6 VRAM)
RAM: SAMSUNG 16GB DDR4-3200
IDE: LM Studio v0.3.9

Deployment Method

0. It is recommended to use an international network environment and enable TUN mode in your proxy tool

1. Download and install the LM Studio client
   Click to go

2. Change the model download directory (recommended)
   This prevents models from consuming too much space on the C drive
   

3. Download the appropriate model based on your computer configuration
   Models with Distill in the name are distilled models
   The recommended model is DeepSeek-R1-Distill-Llama-8B-Abliterated-GGUF, which runs smoothly with 4GB of VRAM. This model also removes some of DeepSeek's built-in restrictions, allowing more freedom in local use
   

4. Load the model
   After the download completes, click the top bar to load the model you just downloaded
   It is recommended to increase GPU Offload and enable Fast Attention for better performance
   

5. Start using it
   

Advanced Usage

Download and Install Cherry Studio

This section will be supplemented with specific download and installation steps later.

Register a SiliconFlow Account

Click to register

Get an API Key

This project trains a small TinyStories-style GPT model from random initialization on an Apple M4 Mac mini 16GB using MLX. It is not about calling APIs or fine-tuning an existing model, but rather walking through the entire pipeline of data preparation, tokenizer, model architecture, training loop, checkpoint, and inference generation.

This write-up leans more toward an engineering retrospective: the focus is not on training a chat-capable model, but on verifying whether a personal machine can complete an end-to-end small-scale LLM training run.

Project repository: sergioperezcheco/llm-from-scratch

Project Results

The final trained model is a GPT with 44M parameters:

| Item | Result |
|

On a Windows Server 2008 EC2 instance, the CloudWatch Agent service status was normal, but monitoring metrics could never be reported to CloudWatch. The logs repeatedly showed x509: certificate signed by unknown authority, and the root cause was ultimately traced to outdated system root certificates and insufficient TLS support.

Symptoms

The CloudWatch Agent logs continuously showed errors like:

WriteToCloudWatch failure, err: RequestError: send request failed
caused by: Post https://monitoring.<region>.amazonaws.com.cn/:
x509: certificate signed by unknown authority

Also note two easily misdiagnosed points:

• ping monitoring.<region>.amazonaws.com.cn failing does not necessarily mean the service is unreachable — Interface Endpoints typically do not respond to ICMP.
• Opening the CloudWatch API endpoint in a browser and getting 404 Not Found is also normal — it is not a regular web service.

Root Cause

The root certificate store on Windows Server 2008 is too old and may lack the root certificates needed to verify AWS server certificates, such as Amazon Root CA 1. Purely internal instances that cannot access the public internet also cannot automatically pull new trusted root certificates.

Additionally, older versions of Windows Server 2008 may also lack patches that support modern TLS chains. The end result is that the CloudWatch Agent cannot complete certificate chain verification when establishing an HTTPS connection.

Resolution Steps

1. Install SHA-2 / TLS Related Patches

First, install the required security patches for Windows Server 2008, such as KB4474419. The system must be restarted after patch installation.

wusa.exe C:\Patches\windows6.1-kb4474419-v3-x64.msu /quiet /norestart
shutdown /r /t 0

2. Import Amazon Root CA 1

Download the Amazon Root CA 1 certificate:

https://www.amazontrust.com/repository/AmazonRootCA1.cer

In an internal network environment, you can first download it from a machine with public internet access, then securely copy it to the instance.

Import it into the trusted root certificate store:

certutil -addstore -f Root C:\Patches\AmazonRootCA1.cer

You can also import it via the certmgr.msc graphical interface into "Trusted Root Certification Authorities."

3. Restart the CloudWatch Agent

net stop "Amazon CloudWatch Agent"
net start "Amazon CloudWatch Agent"

Verification

Check the Agent logs to confirm that x509: certificate signed by unknown authority no longer appears.

You can also test TCP 443 connectivity:

(New-Object System.Net.Sockets.TcpClient).Connect("monitoring.<region>.amazonaws.com.cn", 443)

If the command completes without errors, TCP layer connectivity is confirmed. Ultimately, verify that metrics are being reported normally in the CloudWatch console.

Summary

When running CloudWatch Agent on older Windows Server 2008 systems in a purely internal network environment, the common issue is not VPC Endpoint configuration, but outdated system root certificates and TLS capabilities. The recommended resolution order is:

1. Confirm CloudWatch endpoint TCP 443 is reachable.
2. Install necessary system patches.
3. Manually import Amazon Root CA 1.
4. Restart the CloudWatch Agent and observe the logs.

Such legacy systems should be included in a migration plan, as they will continually encounter certificate, TLS, patch, and software compatibility issues over time.

Symptoms

Executing AWS-RunPatchBaseline scan fails, returning something like:

The find operation did not complete successfully

The HResult may be:

-2145107934

Under the same network environment, Windows Server 2016 works normally while Windows Server 2019 fails.

Troubleshooting Process

1. Verify SSM Endpoints

Test-NetConnection ssm.<region>.amazonaws.com -Port 443
Test-NetConnection ssmmessages.<region>.amazonaws.com -Port 443
Test-NetConnection ec2messages.<region>.amazonaws.com -Port 443

If using China regions or VPC Endpoints, replace with the corresponding domain names.

2. Verify Windows Update Network

Test-NetConnection sls.update.microsoft.com -Port 443
Test-NetConnection download.windowsupdate.com -Port 80

Also check if WSUS is configured:

Get-ItemProperty "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate" -ErrorAction SilentlyContinue
Get-ItemProperty "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU" -ErrorAction SilentlyContinue

3. Export Windows Update Log

Get-WindowsUpdateLog

If the log contains:

The server returned HTTP status code '503'
The service is temporarily overloaded
*FAILED* [80244022] Web service call

This means the Windows Update client has connected to the Microsoft service, but the server returned an unavailable status.

Root Cause

Windows Server 2019 may be accessing a specific Microsoft update service domain, such as:

fe3.delivery.mp.microsoft.com

If that service endpoint is temporarily overloaded, it returns HTTP 503, corresponding to error code:

0x80244022

At the same time, Windows Server 2016 may work fine because it accesses a different set of update service domains that are not affected.

Workarounds

1. Wait for Microsoft Service Recovery

If you've confirmed that the network, SSM, and WSUS configuration are all normal, and the error is clearly 503, the most straightforward approach is to wait for the service to recover and retry.

2. Manually Download Patches

Download the .msu from Microsoft Update Catalog, then install via script or SSM Run Command:

https://www.catalog.update.microsoft.com/

3. Pre-cache Patches

Patch Manager itself does not provide a standard "download only, don't install" mode. You can use a custom SSM document to pre-download patches and execute installation during the maintenance window.

4. Deploy WSUS

For environments with strict patch window requirements, you can deploy an on-premises WSUS server to pre-sync patches locally, reducing dependency on the public Microsoft update service.

Summary

SSM Patch Manager scan failures are not necessarily SSM issues. When troubleshooting, work through the layers:

1. Whether the SSM Agent is online.
2. Whether AWS endpoints are reachable.
3. Whether Windows Update endpoints are reachable.
4. Whether WindowsUpdate.log shows a Microsoft server-side 503.

If the logs clearly show 0x80244022 and HTTP 503, it should generally be treated as a temporary Microsoft update service unavailability — consider retrying, manual patching, pre-caching, or WSUS.

Symptoms

The patch installation phase appears successful, but upon reboot:

We couldn't complete the updates
Undoing changes
Don't turn off your computer

After entering the system, the OS Build has not increased, and reinstalling still results in repeated rollbacks.

WindowsUpdate.log may show:

Post-reboot status ... 0x800f0922

CBS.log contains:

CBS_E_INSTALLERS_FAILED
Per-User Registry Installer ... 0x80070002

CSI logs may also show historical user NTUSER.DAT unload failures.

Troubleshooting Approach

First, rule out common causes:

sfc /scannow
DISM /Online /Cleanup-Image /RestoreHealth

Also check:

• Whether SSU is already installed.
• Whether the C drive has sufficient space.
• Whether failure persists after resetting SoftwareDistribution / catroot2.

If all of these are normal, examine the CBS/CSI logs to see if failures are concentrated in the Per-User Registry Installer phase.

Root Cause

In this case, the issue was concentrated in historical user profiles. The server had multiple legacy user directories, Unknown Profiles, and even abnormally large user profiles. During the reboot phase, updates need to load or unload user registry hives, and some NTUSER.DAT files cannot be properly unloaded, causing the patch transaction to fail and triggering a rollback.

Solution

1. Back Up First

Before performing operations in production, create an AMI or snapshot. User profile cleanup carries data risk and should not be done by blindly deleting in production.

2. Clean Up Unknown Profiles

Via the GUI:

1. Run sysdm.cpl.
2. Go to the "Advanced" tab.
3. Click "Settings" in the "User Profiles" section.
4. Delete profiles with Unknown status or those confirmed to be no longer in use.

3. Clean Up Registry ProfileList If Necessary

Carefully open the registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList

Cross-reference the C:\Users directory with SIDs in ProfileList, and clean up invalid entries. Export a registry backup before proceeding.

4. Reinstall the Patch

After cleanup, reboot, then reinstall the target cumulative update.

5. Temporary Workaround

If profiles cannot be cleaned up immediately and a newer monthly patch has been released, you can test installing the updated cumulative patch directly. Windows cumulative updates typically include the previous month's security content, but this should only be used as a temporary workaround — the underlying issue should still be addressed through cleanup.

Summary

When Windows patches roll back during the reboot phase, don't focus solely on Windows Update. If CBS/CSI logs point to the Per-User Registry Installer and user hive unload failures, focus on examining historical user profiles.

Profile accumulation is common on servers that have been accessed by many users over a long period. It is recommended to regularly clean up obsolete profiles to avoid issues surfacing during patch windows.

Symptoms

Users receive a permission denied message when accessing the SMB share path. The administrator has already added permissions to the target domain group in the "Security" tab, but access still fails.

Key Concepts

SMB share access is controlled by two permission layers simultaneously:

• Share Permissions
• Security Permissions / NTFS Permissions

The effective permission is the intersection of both. If only NTFS permissions are configured without share permissions, users may still be denied.

Troubleshooting Steps

1. Check Share Permissions

Log in to a domain-joined Windows machine with a domain administrator account and open:

compmgmt.msc

Connect to the FSx SVM DNS name, then navigate to:

System Tools -> Shared Folders -> Shares

Find the target share, open its properties, and check the "Share Permissions" tab. Confirm that the target user or group has at least Read permission.

2. Check NTFS Permissions

Then check the "Security" tab to confirm the file system permissions also include the target user or group.

3. Refresh Kerberos Tickets

If a user was just added to a domain group, the Kerberos TGT in their current login session may still contain old group membership information.

The most reliable approach is to have the user fully log off and log back in, rather than just locking the screen or disconnecting RDP.

You can also try purging tickets:

klist purge

But in production troubleshooting, a full logoff and logon is more straightforward and reliable.

Why This Happens

After a Windows user logs in, they receive a Kerberos ticket containing group membership information. If an administrator modifies group memberships while the user is logged in, the user's existing ticket will not automatically reflect the new group membership. FSx still sees the old identity information, causing the permission check to fail.

Summary

When FSx ONTAP SMB share access is denied, investigate in this order:

1. Whether share permissions allow access.
2. Whether NTFS permissions allow access.
3. Whether the user has logged off and back in to refresh Kerberos tickets.

Only checking the "Security" tab is not enough — this is the most common mistake in SMB permission troubleshooting.

Symptoms

FSx creation fails, with the following appearing in the console or error message:

setupFileServerRole failed
Get-ADComputer : Unable to contact the server.
This may be because this server does not exist, it is currently down,
or it does not have the Active Directory Web Services running.

This error typically occurs when FSx is setting up the file server role, joining, or querying AD objects.

Root Cause

Get-ADComputer depends on Active Directory Web Services, i.e., ADWS. ADWS uses:

TCP 9389

For FSx for Windows Single-AZ 2 and Multi-AZ types, FSx needs to access the domain controller's TCP 9389. If this port is blocked by security groups, NACLs, enterprise firewalls, or cross-region network policies, file system creation will fail.

Verification Method

On a domain-joined EC2 Windows instance in the same subnet and same security group as FSx, run the AD validation tool.

Install-WindowsFeature RSAT-AD-PowerShell

Invoke-WebRequest `
  "https://docs.aws.amazon.com/fsx/latest/WindowsGuide/samples/AmazonFSxADValidation.zip" `
  -OutFile "AmazonFSxADValidation.zip"

Expand-Archive -Path "AmazonFSxADValidation.zip"
Import-Module .\AmazonFSxADValidation

$Credential = Get-Credential
$Args = @{
  DomainDNSRoot  = "example.com"
  DnsIpAddresses = @("DC_IP_1", "DC_IP_2")
  SubnetIds      = @("subnet-xxxxxxxx")
  Credential     = $Credential
}

$Result = Test-FSxADConfiguration @Args
$Result.Failures

If the output contains a TCP 9389 failure entry, you can confirm it is an ADWS port connectivity issue.

Why Single-AZ 1 Might Succeed

Single-AZ 1 has different requirements for TCP 9389 and may succeed in the same environment. This can help determine whether service account permissions, DNS, and basic AD ports are functioning properly.

If Single-AZ 1 succeeds but Single-AZ 2 / Multi-AZ fails, the troubleshooting focus should shift to TCP 9389.

Solution

Allow TCP 9389 from the FSx subnet to all domain controllers:

• FSx security group outbound rules.
• Domain controller security group inbound rules.
• Network ACL rules in both directions.
• Local or cross-region firewall policies.
• Intermediate firewalls in enterprise networks.

After allowing access, re-run the validation tool to confirm there are no failures, then recreate FSx.

Summary

When FSx for Windows fails to join a self-managed AD, don't only check common ports like 389, 445, and 88. For Single-AZ 2 and Multi-AZ, TCP 9389 is equally critical.

When you see Get-ADComputer or ADWS-related errors, prioritize verifying 9389 connectivity from the FSx subnet to all DCs.

Prerequisites

Before enabling SQL Server HA license savings, confirm the environment meets the requirements:

• Windows Server 2019 or later.
• SQL Server 2017 or later.
• An HA cluster supports only two EC2 nodes.
• Instances need to run SSM Agent.
• Instance IAM Role needs EC2 SQL HA and SSM-related permissions.

If the environment is still on Windows Server 2016, or the cluster has more than two nodes, the prerequisites for this feature are not met.

Standby Node Restrictions

To qualify for license cost reduction, the standby node must remain passive:

• It does not process incoming business traffic.
• It does not run active SQL Server workloads.
• It cannot serve as a readable secondary replica to handle read queries.
• It should not run standalone databases outside the availability group.

The core determination is simple: as long as the node is providing data services, it is no longer a pure standby.

Readable Secondary Affects Cost Reduction

When Readable Secondary is enabled in an Always On availability group, the secondary replica can be accessed by applications, reports, or manual queries. Under license logic, this constitutes active use and requires full SQL Server licensing.

Therefore, if the goal is to obtain the standby node license cost reduction, do not enable readable secondary replicas.

Does Backup Require a Readable Secondary?

No. SQL Server supports executing certain backup scenarios on non-readable secondary replicas. In other words, for full backups and log backups, there is no need to set the secondary replica as readable.

Before actual configuration, verify against the SQL Server version and availability group backup preferences.

Enablement Steps Overview

1. Confirm SSM Agent Is Online

aws ssm describe-instance-information

The instance should show PingStatus: Online.

2. Configure IAM Permissions

Attach the following to the instance profile:

• AmazonSSMManagedInstanceCore
• AWSEC2SqlHaInstancePolicy

3. Prepare SQL Credentials

By default, NT AUTHORITY\SYSTEM can be used to read SQL Server HA metadata. If the environment restricts this account, place the SQL Server credentials in Secrets Manager and specify them during enablement.

4. Enable in the EC2 Console

In the EC2 console, select the HA cluster-related instances and navigate to:

Actions -> Instance settings -> Modify SQL High Availability settings

Check the prerequisites, and enable license savings after they pass.

After enabling, you should see:

• Primary node: Active / Full license included
• Standby node: Standby / Waived

Summary

SQL Server HA standby node license cost reduction is not a simple toggle. What truly matters is that the standby node must remain passive.

If you enable a readable secondary for queries, reports, or application reads, you lose the cost reduction eligibility. For backup scenarios, prioritize using SQL Server's supported secondary replica backup capabilities rather than turning the standby node into a readable workload node.

Symptoms

After unchecking .NET Framework 4 in Server Manager's "Remove Roles and Features" and rebooting the instance:

• Server Manager won't open.
• PowerShell reports the ServerManager command is not recognized.
• Install-WindowsFeature reports the feature name doesn't exist.
• Some PowerShell management capabilities stop working.

Example error:

ServerManager : The term 'ServerManager' is not recognized

Root Cause

Windows Server's Server Manager and related PowerShell modules depend on NetFx4-OC-Package. Unchecking .NET Framework 4 Features in the GUI actually disables a batch of OC packages that depend on NetFx4.

This is not the same as the .NET Framework 4.8 runtime version. The registry may still show .NET 4.8 is present, but the Windows optional component NetFx4 has been disabled.

Additionally, NetFx3 cannot substitute for NetFx4. Running:

DISM /Online /Enable-Feature /FeatureName:NetFx3 /All

will not restore Server Manager, which depends on .NET 4.

Recovery Steps

Before proceeding, it is recommended to create an AMI backup. All commands should be run as administrator.

1. Enable NetFx4

DISM /Online /Enable-Feature /FeatureName:NetFx4 /All

2. Enable Server Manager GUI Management Components

DISM /Online /Enable-Feature /FeatureName:Server-Gui-Mgmt /All

3. Reboot the Instance

shutdown /r /t 0

Verify after rebooting.

4. Restore Other Dependent Components as Needed

If your workload uses IIS, WCF, PowerShell ISE, or DSC, enable them as needed:

DISM /Online /Enable-Feature /FeatureName:NetFx4Extended-ASPNET45 /All
DISM /Online /Enable-Feature /FeatureName:WCF-HTTP-Activation45 /All
DISM /Online /Enable-Feature /FeatureName:WCF-TCP-PortSharing45 /All
DISM /Online /Enable-Feature /FeatureName:IIS-ASPNET45 /All
DISM /Online /Enable-Feature /FeatureName:IIS-NetFxExtensibility45 /All
DISM /Online /Enable-Feature /FeatureName:MicrosoftWindowsPowerShellISE /All
DISM /Online /Enable-Feature /FeatureName:DSC-Service /All

Verification

ServerManager
Get-Command Install-WindowsFeature
DISM /Online /Get-Features | findstr /I "NetFx4 Server-Gui PowerShell"

Summary

To meet .NET version compliance requirements, the correct approach is to install .NET cumulative updates, not to disable .NET Framework 4 Features. The latter does not make .NET "safely disappear" from the system — instead, it breaks the Windows Server management toolchain.

During recovery, enabling NetFx4 alone is not enough — you must also enable Server-Gui-Mgmt and reboot.

Risk Points

If you enable BitLocker directly on the C drive and reboot, you may encounter:

• The instance status shows running, but RDP cannot connect.
• The boot phase is waiting for a BitLocker password or recovery key.
• The console shows a black screen or cannot accept input.
• The workload is unavailable for an extended period, and recovery is only possible through snapshot/AMI rollback.

Therefore, you must first verify that the EC2 serial console is available.

Pre-Operation Preparation

• Create an AMI or EBS snapshot.
• Perform a complete rehearsal on a test instance.
• Record and securely store the recovery key offline.
• Confirm acceptance of a reboot and brief downtime window.

Enable Serial Console SAC

Run the following in an administrator PowerShell session:

bcdedit /ems '{current}' on
bcdedit /emssettings EMSPORT:1 EMSBAUDRATE:115200

bcdedit /set '{bootmgr}' displaybootmenu yes
bcdedit /set '{bootmgr}' timeout 15
bcdedit /set '{bootmgr}' bootems yes

shutdown -r -t 0

After rebooting, confirm through the EC2 console via "Connect -> EC2 Serial Console" that you can access the boot interface. If the serial console is not available, do not proceed with encrypting the system drive.

Install BitLocker Feature

Add BitLocker via Server Manager, or use PowerShell:

Install-WindowsFeature BitLocker -IncludeAllSubFeature -IncludeManagementTools
Restart-Computer

Configure Boot Without TPM

Run gpedit.msc and navigate to:

Computer Configuration -> Administrative Templates -> Windows Components -> BitLocker Drive Encryption -> Operating System Drives

Enable "Require additional authentication at startup" and check "Allow BitLocker without a compatible TPM".

Encrypt Data Drives

For data drives, test first:

1. Right-click the data drive and enable BitLocker.
2. Choose password unlock.
3. Save the recovery key.
4. Configure auto-unlock after encryption completes.

Without auto-unlock, the workload may become unavailable after every reboot because the data drive is locked.

Encrypt the System Drive

Enable BitLocker on the C drive, choose password unlock, save the recovery key, and run the BitLocker system check.

After rebooting:

1. RDP is temporarily unavailable.
2. Connect to the EC2 serial console.
3. At the black screen, enter the BitLocker password and press Enter.
4. Windows unlocks and continues booting.
5. Restore RDP after the system finishes booting.

Summary

The key to using BitLocker on EC2 Windows is not clicking "Enable Encryption" — it's whether you can unlock during the boot phase.

You must:

• Create an AMI/snapshot before encrypting.
• Enable and verify the EC2 serial console first.
• Save the recovery key.
• Configure auto-unlock for data drives.
• Rehearse the complete reboot process in a test environment first.

If you only need static EBS encryption in the cloud, prioritize using EBS encryption. BitLocker is more suitable for BYOL or specific compliance scenarios.

Symptoms

• Instance is in running state.
• System status checks and instance status checks pass.
• Security group allows port 3389.
• Instance responds to ping.
• RDP cannot connect.
• Stop & Start resolves the issue.

This behavior closely resembles a network issue, but the logs may point to insufficient system resources.

Key Logs

The event log may contain:

Not enough storage is available to process this command.

And:

System.OutOfMemoryException

You may also see Winlogon-related events, such as Winlogon crashing or failing to create a login session.

Here, "storage" does not necessarily refer to disk — in the context of Windows error codes, it can also mean insufficient memory or system resources.

Root Cause

When instance memory and page file resources are exhausted, the system cannot allocate resources for critical processes. Components required for RDP login — such as Winlogon, LSASS, and Remote Desktop Services — may fail to function properly.

Stop & Start clears the memory state, so the issue temporarily disappears, but if the instance size or application memory usage is not addressed, the problem will recur.

Troubleshooting Direction

1. Check Event Logs

Focus on the period around the failure in:

• Application.evtx
• System.evtx
• Setup.evtx

Look for:

• OutOfMemoryException
• Not enough storage is available
• Winlogon errors
• Security software or monitoring agent anomalies
• Windows Update-related anomalies

2. Check Instance Size

Confirm whether the current instance memory meets peak workload demands. If it is consistently near the upper limit, upgrade the instance size or optimize the application.

3. Deploy OS Metrics Monitoring

CloudWatch does not collect Windows memory metrics by default. You need to install the CloudWatch Agent to collect:

• Memory utilization
• Pagefile utilization
• Disk utilization
• Key process metrics

And set up alarms, such as alerting when memory utilization exceeds 85%.

Recommended Actions

Short-Term Recovery

Stop & Start can release memory and temporarily restore login capability. But this is not a permanent fix.

Medium-Term Optimization

Investigate applications, monitoring agents, and security software that consume high memory to determine if there are memory leaks or overly heavy configurations.

Long-Term Solution

If peak workloads genuinely require more memory, upgrade to a larger instance size. Before upgrading, create an AMI and confirm instance type compatibility with ENA/NVMe.

Summary

RDP login failure is not necessarily a port 3389, security group, or NACL issue. As long as the instance is still pingable and status checks pass, you should also examine the Windows event logs.

If the logs show OOM, insufficient system resources, and Winlogon anomalies, the root cause is most likely memory exhaustion. Stop & Start only provides temporary recovery — long-term, you need to monitor memory and adjust instance size or application configuration.

Symptoms

Client access attempt:

\\<server-private-ip>\share

Cannot open, and no authentication prompt appears. Command line execution:

net use Z: \\<server-private-ip>\share

Reports:

System error 1792 has occurred.
The attempt to logon to the network account failed because the network logon service is not started.

Afterward, port 445 connection timeouts and ping failures may even occur.

Troubleshooting Process

1. Rule Out AWS Network Issues First

Confirm:

• Both instances are in the same VPC or have routable connectivity.
• Security group allows TCP 445.
• NACL is not blocking.
• Route table has a local or correct route.
• Windows Defender Firewall policy is not blocking.

2. Packet Capture on Both Ends

Packet capture shows the SMB protocol has already begun negotiation:

SMB2 Negotiate Protocol Request
SMB2 Negotiate Protocol Response
SMB2 Session Setup Request
SMB2 Session Setup Response: STATUS_NETLOGON_NOT_STARTED

This indicates traffic has reached the server — it is not an AWS underlying network packet drop.

3. Understanding Netlogon

In a Workgroup environment, the Netlogon service not running by default is normal. Local account SMB authentication typically completes through local SAM + NTLM, and should not necessarily fail just because Netlogon is stopped.

If STATUS_NETLOGON_NOT_STARTED is returned and then all traffic is blocked, suspect that security software or EDR is intercepting SMB authentication traffic.

Solution

1. Temporarily Disable EDR to Verify

Temporarily disable security software during a change window to verify whether SMB access is restored. If port 445 and ping both recover after disabling, the root cause is essentially confirmed.

2. Use Explicit Local Account Authentication

In a Workgroup environment, do not rely on implicit credentials. Use a server local account:

net use Z: \\<server-private-ip>\share /user:<server-computer-name>\Administrator

Note that <server-computer-name> must be the server's computer name, not the client's.

3. Adjust EDR Policy

Contact the security software vendor or security team to whitelist normal SMB/NTLM authentication traffic and prevent false positives.

4. Long-Term Recommendation: Join a Domain

If multiple Windows instances frequently share files, consider joining Active Directory and using domain accounts with group-based permissions to reduce the complexity of Workgroup + local account authentication.

Summary

When EC2 Windows SMB access fails, don't only look at security groups. If packet capture already shows SMB protocol negotiation and authentication-stage errors, the issue has moved into the OS or security software layer.

STATUS_NETLOGON_NOT_STARTED in a Workgroup scenario does not necessarily mean Netlogon itself is the root cause. Combined with subsequent traffic being blocked, EDR or security software interception should be a primary investigation focus.

Why VM Export Fails

Running create-instance-export-task may produce:

An error occurred (NotExportable) when calling the CreateInstanceExportTask operation:
The image ID (ami-xxxxxxxx) provided contains AWS-licensed software and is not exportable.

This is a product limitation, not an IAM permissions issue. Windows / SQL Server / Marketplace images launched from public AWS AMIs are generally non-exportable.

Alternatives

Available options:

• If you can log in to the instance: use Disk2vhd.
• If you don't want to spin up a rescue instance: use coldsnap + qemu-img.
• If the instance can be stopped or has crashed: use a rescue instance for block-level EBS reading, then qemu-img conversion.

This article documents the third approach.

Key Pitfall: Storage Controller Drivers

EC2 Windows typically runs on NVMe devices. After exporting to VMware, the controller may change to LSI Logic SAS, SATA, or IDE. If these drivers are not set to Boot start in the Windows registry, you'll get a blue screen at boot:

UNMOUNTABLE_BOOT_VOLUME

So before conversion, you need to offline-modify the SYSTEM hive and set the Start value of common storage drivers to 0.

Procedure

1. Stop Instance and Detach Volume

aws ec2 stop-instances --instance-ids <instance-id>
aws ec2 wait instance-stopped --instance-ids <instance-id>

aws ec2 detach-volume --volume-id <volume-id>
aws ec2 wait volume-available --volume-ids <volume-id>

aws ec2 attach-volume \
  --volume-id <volume-id> \
  --instance-id <helper-instance-id> \
  --device /dev/sdg

It's recommended to create a snapshot of the volume before proceeding.

2. Identify Disk on Rescue Instance

sudo lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT,FSTYPE

On Nitro instances, EBS appears as /dev/nvmeXn1. You can match volume IDs via the SERIAL field.

3. Mount Working Disk

sudo mkfs.xfs -f /dev/nvme1n1
sudo mkdir -p /mnt/work
sudo mount /dev/nvme1n1 /mnt/work

The working disk must be larger than the actual output size of the target volume.

4. Install Tools

sudo dnf install -y qemu-img gcc make perl

If hivex is not available in the repository, you'll need to install it from source. It's used to edit Windows registry hives.

5. Mount NTFS and Backup SYSTEM Hive

sudo modprobe ntfs3
sudo mkdir -p /mnt/windows
sudo mount -t ntfs3 -o rw /dev/nvme2n1p1 /mnt/windows

sudo cp /mnt/windows/Windows/System32/config/SYSTEM /mnt/work/SYSTEM_BACKUP
sudo cp /mnt/windows/Windows/System32/config/SYSTEM /tmp/SYSTEM_EDIT

6. Enable Generic Storage Drivers

The following services need their Start set to 0:

| Service | Purpose |
|

Project repository: bin456789/reinstall

Reference documentation: CSDN original article

Steps

Why Graviton Can't Officially Run Windows

AWS only provides Windows AMIs for the x86 architecture. Graviton uses the ARM64 architecture. Although Microsoft has Windows on ARM, AWS has not done official driver adaptation and AMI publishing for the Graviton platform. So the normal flow is: choose t3/t3a x86 instances for Windows, and Graviton can only run Linux.

But "not officially supported" doesn't mean "can't run." The reinstall.sh script can directly DD-install Windows on an existing Linux instance, including the ARM64 version. Test environment:

• Instance type: t4g.large (2 vCPU / 8 GB)
• Original system: Amazon Linux 2023 aarch64, 50 GB EBS (gp3), UEFI boot
• Target system: Windows 11 Pro 25H2 ARM64 (Build 26200.6584), Chinese version

Core Finding: The Counter-Intuitive NVMe Driver Pitfall

The most counter-intuitive part of the entire process: Injecting the official AWS NVMe driver actually causes boot failure; only the Microsoft inbox StorNVMe works properly.

StorNVMe.sys is a standard NVMe driver built into the system image by Microsoft since Windows 8.1. It loads automatically during installation without any external injection or manual operation. The Has StorNVMe: true in the reinstall.sh log means the script detected the ISO includes this inbox driver.

Real-world comparison:

| Approach | NVMe Driver | ENA Driver | Result |
|

Symptoms

Accessing an internal service results in an error:

curl https://service.example.internal/api/v1/health

curl: (60) SSL certificate problem: unable to get local issuer certificate

The architecture is roughly:

Client -> Internal ALB 443 -> Backend Target 443

If bypassing the ALB and directly accessing the backend target succeeds, it indicates the backend service itself is most likely not the root cause.

Troubleshooting Approach

1. Don't Rely on Public SSL Check Results

If the domain resolves to an internal ALB in a Route 53 private hosted zone, a public SSL checker may see a different public DNS record. Public check results cannot represent the certificate actually served by the internal ALB.

2. Use openssl to Inspect the Actual ALB Certificate

Run the following on an internal client:

openssl s_client -showcerts \
  -connect <alb-dns-name>:443 \
  -servername service.example.internal </dev/null

Focus on two things:

• Whether the certificate SAN/CN covers the accessed domain.
• Whether the intermediate certificate served by the ALB matches the site certificate's signing chain.

3. Distinguish Between Two Problems

A broken certificate chain and a domain mismatch are two different problems:

• A broken chain causes the client to fail to find a trusted parent CA.
• SAN not covering the domain causes hostname verification to fail.

Either one can cause HTTPS to fail.

Root Cause

In this case, the site certificate was issued by a DV intermediate CA, but the one included during ACM import was a different OV intermediate CA. The ALB therefore served a mismatched intermediate certificate chain to the client, and curl could not build a complete trust chain.

At the same time, the ALB-bound certificate's SAN also did not cover the actual domain being accessed. In other words, even if the certificate chain were fixed, it would still fail due to the domain mismatch.

Solution

1. Fix the ACM Certificate Chain

Re-import the original ACM certificate and upload the correct intermediate certificate chain:

aws acm import-certificate \
  --certificate-arn <certificate-arn> \
  --certificate fileb://site.crt \
  --private-key fileb://site.key \
  --certificate-chain fileb://correct-chain.crt

The advantage of using reimport is that it preserves the original ARN, so the ALB listener does not need to re-select a certificate.

2. Ensure the Certificate Covers the Accessed Domain

Check whether the certificate SAN includes the domain being used:

openssl x509 -in site.crt -noout -text | grep -A1 "Subject Alternative Name"

If it does not, you need to request or import a new certificate that covers the target domain and bind it to the ALB HTTPS listener.

3. Clean Up ALB Listener Certificates

If the ALB has multiple certificates attached, it is recommended to clean up unused certificates to avoid SNI matching confusion and operational judgment errors.

Summary

When ALB HTTPS reports unable to get local issuer certificate, the troubleshooting focus should be the certificate chain actually served by the ALB, not the backend Nginx or public SSL check results.

Recommended fixed troubleshooting order:

1. Use openssl s_client -showcerts on an internal client to inspect the certificate served by the ALB.
2. Check whether SAN/CN covers the accessed domain.
3. Check whether the intermediate certificate matches the site certificate's signing chain.
4. Use ACM reimport to fix the certificate chain, and replace the listener certificate if necessary.

Scenario

The goal is to count the following in a versioning-enabled S3 bucket:

• Number of objects with a specified file extension.
• Total size of objects with a specified file extension.
• Including all historical versions, not just the current version.

If the business requires "results right now," S3 Inventory may not be suitable because it is an asynchronous report and the first generation typically has a delay.

Why Use list-object-versions

The regular list-objects only looks at the current object version and cannot cover historical versions. For buckets with versioning enabled, use:

aws s3api list-object-versions

Then use --query 'Versions[*].[Key, Size]' to extract only the object Key and Size, reducing downstream processing cost.

Counting Command

The following example counts .jpg and .png objects:

aws s3api list-object-versions \
  --bucket <bucket-name> \
  --region <region> \
  --query 'Versions[*].[Key, Size]' \
  --output text |
grep -Ei "\.(jpg|png)[[:space:]]+[0-9]+$" |
awk '
BEGIN {
  fmt = "Total image objects (including historical versions): %d\n"
}
{
  count++;
  size += $NF;
}
END {
  print "======================";
  printf fmt, count;
  printf "Total size (including historical versions): %.2f GB\n", size/1024/1024/1024;
  print "======================";
}'

Example output:

======================
Total image objects (including historical versions): 120446
Total size (including historical versions): 56.33 GB
======================

Notes

• It is recommended to execute this on an EC2 instance in the same region to reduce network latency.
• If there are many objects, CLI calls will incur List request costs.
• If you only need the current version, do not use list-object-versions — use list-objects-v2 instead.
• If the object scale is very large and some delay is acceptable, S3 Inventory is more suitable for periodic reporting.
• If keys contain special characters such as newlines, text pipeline processing will have edge-case issues. For rigorous scenarios, use JSON + jq.

Summary

When you need to urgently count objects with specific extensions in an S3 bucket, list-object-versions + grep + awk is a simple and effective solution. Its advantages are that it is real-time, lightweight, and requires no waiting for Inventory; its disadvantage is that it is more suited for one-off counting, not long-term periodic reporting.

Scenario

In an enterprise environment, many domain users log into the AWS console via SAML AssumeRole, potentially using high-privilege roles like PowerUser or Administrator.

Due to compliance requirements, certain users need to be restricted from accessing EC2 instances through the console, primarily involving two capabilities:

• SSM Session Manager: ssm:StartSession
• EC2 Instance Connect: ec2-instance-connect:SendSSHPublicKey

Why Not Use RoleSessionName

After SAML federated login, IAM generates a session identifier. The more stable and available global condition key for policy evaluation is aws:userid. It typically contains the role ID and session name, in a format like:

<role-id>:<session-name>

Because the role ID portion changes with the Role, it is not suitable for hardcoding. Instead, you can use a wildcard to match the session suffix:

*:user-or-ou-id

Example Policy

Add an explicit Deny to the target IAM Role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyEC2LoginForSpecificFederatedUsers",
      "Effect": "Deny",
      "Action": [
        "ssm:StartSession",
        "ec2-instance-connect:SendSSHPublicKey"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:userid": [
            "*:<user-or-ou-id>"
          ]
        }
      }
    }
  ]
}

Explicit Deny takes priority over Allow, so even if the Role already has high-privilege policies, users matching the condition will still be blocked.

Verification Method

1. Log into the console as the target SAML user.
2. Attempt to connect to EC2 via Session Manager.
3. Attempt to use EC2 Instance Connect.
4. Then verify that unrestricted users are not affected.

If the Deny is working, the target user will be denied when calling the related APIs.

Recommended Long-Term Approach

Hardcoding user suffixes based on aws:userid can quickly solve the problem, but the maintenance cost is high. A better approach is to split permissions at the identity source level:

• Create a restricted user group in AD, e.g., AWS-No-EC2-Login.
• Map different user groups to different IAM Roles in the IdP.
• Create a dedicated restricted Role that does not grant EC2 login capabilities.

This makes permission boundaries clearer and easier to audit and automate.

Summary

If you need to temporarily restrict certain SAML users from logging into EC2 via the console, you can use aws:userid + explicit Deny to precisely block ssm:StartSession and ec2-instance-connect:SendSSHPublicKey.

In the long term, it is recommended to manage user grouping and Role mapping at the identity source side, rather than maintaining increasingly complex conditional policies within a single high-privilege Role.