[{"data":1,"prerenderedAt":2911},["ShallowReactive",2],{"blog-layout-\u002Fblog\u002Fhomelab-flux-gitops-journey":3,"blog-\u002Fblog\u002Fhomelab-flux-gitops-journey":1719},{"id":4,"title":5,"body":6,"date":1710,"description":1711,"extension":1712,"image":1713,"meta":1714,"navigation":846,"path":1715,"seo":1716,"stem":1717,"__hash__":1718},"blog\u002Fblog\u002Fhomelab-flux-gitops-journey.md","Building a Homelab with Flux GitOps: Lessons from Three Months",{"type":7,"value":8,"toc":1683},"minimark",[9,13,16,19,24,70,74,79,90,93,141,150,154,157,172,178,182,185,195,214,292,311,316,320,323,351,429,435,467,474,478,485,495,566,572,576,580,587,680,683,687,690,712,718,774,777,780,784,787,891,895,902,928,938,989,1000,1134,1137,1146,1150,1153,1179,1193,1204,1218,1222,1229,1236,1249,1256,1261,1294,1305,1309,1312,1316,1326,1337,1341,1350,1395,1405,1410,1565,1569,1575,1579,1582,1603,1607,1610,1627,1630,1634,1640,1643,1647,1650,1664,1667,1670,1679],[10,11,12],"p",{},"I've been running a homelab for a few years, but this March I decided to do it properly. Talos Linux on a Mac Mini, Flux for GitOps, SOPS for secrets, and a homepage dashboard that shows everything at a glance.",[10,14,15],{},"Three months in, I have a working cluster with Immich, Plex, Grafana, CrowdSec, Traefik, and a handful of other services. More importantly, I have documentation that actually compounds — each problem I solved made the next one easier.",[10,17,18],{},"Here's what I built, what broke, and how I'll do it again.",[20,21,23],"h2",{"id":22},"the-stack","The Stack",[25,26,27,35,41,47,53,64],"ul",{},[28,29,30,34],"li",{},[31,32,33],"strong",{},"Talos Linux"," — Immutable, API-driven Kubernetes. No SSH, no shell, no apt-get. Sounds limiting until you realize you never need to debug \"what changed on the node.\"",[28,36,37,40],{},[31,38,39],{},"Flux"," — GitOps controller. Everything lives in a git repo, Flux reconciles it to the cluster. Push a change, it lands. Revert a change, it's gone.",[28,42,43,46],{},[31,44,45],{},"SOPS + Age"," — Encrypted secrets in git. No plaintext, no sealed-secrets complexity. Decrypt with your age key, apply, done.",[28,48,49,52],{},[31,50,51],{},"Homepage"," — Dashboard with widgets for every service. Internal URLs, API tokens, done.",[28,54,55,58,59,63],{},[31,56,57],{},"Traefik"," — Ingress with ACME DNS challenges for ",[60,61,62],"code",{},"*.example.com"," certs.",[28,65,66,69],{},[31,67,68],{},"CrowdSec"," — Bouncer + LAPI for intrusion detection at the edge.",[20,71,73],{"id":72},"the-journey-aka-things-that-broke","The Journey (aka Things That Broke)",[75,76,78],"h3",{"id":77},"storage-classes-matter-more-than-you-think","Storage Classes Matter More Than You Think",[10,80,81,82,85,86,89],{},"Early mistake: I used NFS (",[60,83,84],{},"nfs-synology",") for everything. Loki crashed in a persistent loop with ",[60,87,88],{},"directory not empty"," errors. PostgreSQL had WAL corruption.",[10,91,92],{},"The problem: NFS lacks POSIX fsync\u002Flocking semantics. Databases need them.",[94,95,96,112],"table",{},[97,98,99],"thead",{},[100,101,102,106,109],"tr",{},[103,104,105],"th",{},"Storage Class",[103,107,108],{},"Use For",[103,110,111],{},"Don't Use For",[113,114,115,129],"tbody",{},[100,116,117,123,126],{},[118,119,120],"td",{},[60,121,122],{},"local-path",[118,124,125],{},"Databases, WAL, TSDB",[118,127,128],{},"—",[100,130,131,135,138],{},[118,132,133],{},[60,134,84],{},[118,136,137],{},"Photo libraries, bulk storage",[118,139,140],{},"Anything with strict fsync",[10,142,143,146,147,149],{},[31,144,145],{},"Lesson",": ",[60,148,122],{}," for databases, NFS for everything else. One decision fixed a week of crash loops.",[75,151,153],{"id":152},"node-scheduling-has-layers","Node Scheduling Has Layers",[10,155,156],{},"The cluster went through four scheduling strategies:",[158,159,160,163,166,169],"ol",{},[28,161,162],{},"Worker node only — simple, single point of failure",[28,164,165],{},"All nodes with control-plane tolerations — apps ran on control plane, messy",[28,167,168],{},"Worker-only apps — clean isolation",[28,170,171],{},"Monitoring on control plane — freed worker resources when I ran out",[10,173,174,177],{},[31,175,176],{},"Current rule",": Apps on workers. Monitoring on control plane. Documented exceptions in HelmRelease comments.",[75,179,181],{"id":180},"helm-upgrades-the-ssa-trap","Helm Upgrades: The SSA Trap",[10,183,184],{},"Flux HelmRelease upgrades failed with:",[186,187,192],"pre",{"className":188,"code":190,"language":191},[189],"language-text","invalid operation: cannot use force conflicts and force replace together\n","text",[60,193,190],{"__ignoreMap":194},"",[10,196,197,198,201,202,205,206,209,210,213],{},"Turns out ",[60,199,200],{},"install.serverSideApply"," is a boolean (",[60,203,204],{},"false","), but ",[60,207,208],{},"upgrade.serverSideApply"," is an enum string (",[60,211,212],{},"\"disabled\"","). Same field name, different types.",[186,215,219],{"className":216,"code":217,"language":218,"meta":194,"style":194},"language-yaml shiki shiki-themes material-theme-lighter github-light github-dark monokai","spec:\n  install:\n    serverSideApply: false   # boolean\n  upgrade:\n    serverSideApply: disabled  # enum string\n    force: true\n","yaml",[60,220,221,234,242,259,267,281],{"__ignoreMap":194},[222,223,226,230],"span",{"class":224,"line":225},"line",1,[222,227,229],{"class":228},"sHsBP","spec",[222,231,233],{"class":232},"swvn1",":\n",[222,235,237,240],{"class":224,"line":236},2,[222,238,239],{"class":228},"  install",[222,241,233],{"class":232},[222,243,245,248,251,255],{"class":224,"line":244},3,[222,246,247],{"class":228},"    serverSideApply",[222,249,250],{"class":232},":",[222,252,254],{"class":253},"s8HiA"," false",[222,256,258],{"class":257},"ss7Ak","   # boolean\n",[222,260,262,265],{"class":224,"line":261},4,[222,263,264],{"class":228},"  upgrade",[222,266,233],{"class":232},[222,268,270,272,274,278],{"class":224,"line":269},5,[222,271,247],{"class":228},[222,273,250],{"class":232},[222,275,277],{"class":276},"sLACW"," disabled",[222,279,280],{"class":257},"  # enum string\n",[222,282,284,287,289],{"class":224,"line":283},6,[222,285,286],{"class":228},"    force",[222,288,250],{"class":232},[222,290,291],{"class":253}," true\n",[10,293,294,295,298,299,302,303,306,307,310],{},"The OpenTelemetry operator upgrade (0.109 → 0.110) removed ",[60,296,297],{},"kube-rbac-proxy",", changing Service port layouts. Three-way merge created duplicate ",[60,300,301],{},"metrics"," ports. Fixed by pinning the version and using ",[60,304,305],{},"force: true"," + ",[60,308,309],{},"serverSideApply: disabled",".",[10,312,313,315],{},[31,314,145],{},": Check chart changelogs for breaking infrastructure changes. Stay one version behind if unsure.",[75,317,319],{"id":318},"synology-traefik-special-handling","Synology + Traefik = Special Handling",[10,321,322],{},"Getting Traefik to proxy Synology DSM\u002FDrive required three specific fixes:",[158,324,325,331,337],{},[28,326,327,330],{},[31,328,329],{},"No CrowdSec on Synology routes"," — DSM's WebSocket-based UI and asset loading break with request inspection middleware",[28,332,333,336],{},[31,334,335],{},"Disable HTTP\u002F2 upstream"," — Some DSM builds have HTTP\u002F2 bugs causing connection failures",[28,338,339,342,343,346,347,350],{},[31,340,341],{},"EndpointSlice-only"," — Mixing v1 ",[60,344,345],{},"Endpoints"," with ",[60,348,349],{},"EndpointSlice"," causes ~50% 502s when IPs differ",[186,352,354],{"className":216,"code":353,"language":218,"meta":194,"style":194},"apiVersion: traefik.io\u002Fv1alpha1\nkind: ServersTransport\nmetadata:\n  name: synology-insecure\n  namespace: traefik\nspec:\n  insecureSkipVerify: true\n  disableHTTP2: true\n",[60,355,356,366,376,383,393,403,409,419],{"__ignoreMap":194},[222,357,358,361,363],{"class":224,"line":225},[222,359,360],{"class":228},"apiVersion",[222,362,250],{"class":232},[222,364,365],{"class":276}," traefik.io\u002Fv1alpha1\n",[222,367,368,371,373],{"class":224,"line":236},[222,369,370],{"class":228},"kind",[222,372,250],{"class":232},[222,374,375],{"class":276}," ServersTransport\n",[222,377,378,381],{"class":224,"line":244},[222,379,380],{"class":228},"metadata",[222,382,233],{"class":232},[222,384,385,388,390],{"class":224,"line":261},[222,386,387],{"class":228},"  name",[222,389,250],{"class":232},[222,391,392],{"class":276}," synology-insecure\n",[222,394,395,398,400],{"class":224,"line":269},[222,396,397],{"class":228},"  namespace",[222,399,250],{"class":232},[222,401,402],{"class":276}," traefik\n",[222,404,405,407],{"class":224,"line":283},[222,406,229],{"class":228},[222,408,233],{"class":232},[222,410,412,415,417],{"class":224,"line":411},7,[222,413,414],{"class":228},"  insecureSkipVerify",[222,416,250],{"class":232},[222,418,291],{"class":253},[222,420,422,425,427],{"class":224,"line":421},8,[222,423,424],{"class":228},"  disableHTTP2",[222,426,250],{"class":232},[222,428,291],{"class":253},[10,430,431,434],{},[31,432,433],{},"Cluster hygiene tip:"," If you ever have ~50% 502s on a route, check for mixed Endpoints + EndpointSlice:",[186,436,440],{"className":437,"code":438,"language":439,"meta":194,"style":194},"language-bash shiki shiki-themes material-theme-lighter github-light github-dark monokai","kubectl get endpoints -n traefik synology-dsm  # Delete if present\n","bash",[60,441,442],{"__ignoreMap":194},[222,443,444,448,451,454,458,461,464],{"class":224,"line":225},[222,445,447],{"class":446},"sR7ES","kubectl",[222,449,450],{"class":276}," get",[222,452,453],{"class":276}," endpoints",[222,455,457],{"class":456},"sFhLe"," -n",[222,459,460],{"class":276}," traefik",[222,462,463],{"class":276}," synology-dsm",[222,465,466],{"class":257},"  # Delete if present\n",[10,468,469,470,473],{},"Manually created Endpoints spawn a mirrored EndpointSlice via ",[60,471,472],{},"endpointslicemirroring-controller",". Delete the Endpoints object and rely on Git-managed EndpointSlice only.",[75,475,477],{"id":476},"the-voidauth-hairpin-coredns-split-horizon","The VoidAuth Hairpin (CoreDNS Split-Horizon)",[10,479,480,481,484],{},"Initial VoidAuth setup had 700-900ms latency on unauthenticated requests. Root cause: ",[60,482,483],{},"APP_URL=https:\u002F\u002Fauth.example.com"," caused internal requests to hairpin NAT — out to the public IP and back.",[10,486,487,490,491,494],{},[31,488,489],{},"Fix: CoreDNS split-horizon"," to resolve ",[60,492,493],{},"auth.example.com"," to the Traefik ClusterIP internally:",[186,496,498],{"className":216,"code":497,"language":218,"meta":194,"style":194},"# infrastructure\u002Fconfigs\u002Fcoredns.yaml\ndata:\n  Corefile: |\n    .:53 {\n        # ... other config ...\n        template IN A auth.example.com {\n            answer \"{{ .Name }} 30 IN A 10.101.208.179\"  # Traefik ClusterIP\n        }\n        forward . \u002Fetc\u002Fresolv.conf\n        # ...\n    }\n",[60,499,500,505,512,523,528,533,538,543,548,554,560],{"__ignoreMap":194},[222,501,502],{"class":224,"line":225},[222,503,504],{"class":257},"# infrastructure\u002Fconfigs\u002Fcoredns.yaml\n",[222,506,507,510],{"class":224,"line":236},[222,508,509],{"class":228},"data",[222,511,233],{"class":232},[222,513,514,517,519],{"class":224,"line":244},[222,515,516],{"class":228},"  Corefile",[222,518,250],{"class":232},[222,520,522],{"class":521},"sRxSC"," |\n",[222,524,525],{"class":224,"line":261},[222,526,527],{"class":276},"    .:53 {\n",[222,529,530],{"class":224,"line":269},[222,531,532],{"class":276},"        # ... other config ...\n",[222,534,535],{"class":224,"line":283},[222,536,537],{"class":276},"        template IN A auth.example.com {\n",[222,539,540],{"class":224,"line":411},[222,541,542],{"class":276},"            answer \"{{ .Name }} 30 IN A 10.101.208.179\"  # Traefik ClusterIP\n",[222,544,545],{"class":224,"line":421},[222,546,547],{"class":276},"        }\n",[222,549,551],{"class":224,"line":550},9,[222,552,553],{"class":276},"        forward . \u002Fetc\u002Fresolv.conf\n",[222,555,557],{"class":224,"line":556},10,[222,558,559],{"class":276},"        # ...\n",[222,561,563],{"class":224,"line":562},11,[222,564,565],{"class":276},"    }\n",[10,567,568,571],{},[31,569,570],{},"Result:"," p99 latency reduced from 1.2s → ~300ms (3-4x faster).",[20,573,575],{"id":574},"what-actually-worked","What Actually Worked",[75,577,579],{"id":578},"sops-for-secrets","SOPS for Secrets",[10,581,582,583,586],{},"SOPS with Age keys is dead simple. One gotcha: there's no ",[60,584,585],{},"sops --delete"," command. The workaround:",[186,588,590],{"className":437,"code":589,"language":439,"meta":194,"style":194},"# Decrypt\nSOPS_AGE_KEY_FILE=~\u002F.config\u002Fsops\u002Fage\u002Fkeys.txt sops -d file.sops.yaml > \u002Ftmp\u002Fplain.yaml\n# Edit\nvim \u002Ftmp\u002Fplain.yaml\n# Re-encrypt (filename must match .sops.yaml path regex)\ncp \u002Ftmp\u002Fplain.yaml \u002Ftmp\u002Fplain.yaml.sops.yaml\nSOPS_AGE_KEY_FILE=~\u002F.config\u002Fsops\u002Fage\u002Fkeys.txt sops --config .sops.yaml -e \u002Ftmp\u002Fplain.yaml.sops.yaml > file.sops.yaml\n",[60,591,592,597,625,630,637,642,653],{"__ignoreMap":194},[222,593,594],{"class":224,"line":225},[222,595,596],{"class":257},"# Decrypt\n",[222,598,599,603,607,610,613,616,619,622],{"class":224,"line":236},[222,600,602],{"class":601},"ss--_","SOPS_AGE_KEY_FILE",[222,604,606],{"class":605},"sGXK2","=",[222,608,609],{"class":276},"~\u002F.config\u002Fsops\u002Fage\u002Fkeys.txt",[222,611,612],{"class":446}," sops",[222,614,615],{"class":456}," -d",[222,617,618],{"class":276}," file.sops.yaml",[222,620,621],{"class":605}," >",[222,623,624],{"class":276}," \u002Ftmp\u002Fplain.yaml\n",[222,626,627],{"class":224,"line":244},[222,628,629],{"class":257},"# Edit\n",[222,631,632,635],{"class":224,"line":261},[222,633,634],{"class":446},"vim",[222,636,624],{"class":276},[222,638,639],{"class":224,"line":269},[222,640,641],{"class":257},"# Re-encrypt (filename must match .sops.yaml path regex)\n",[222,643,644,647,650],{"class":224,"line":283},[222,645,646],{"class":446},"cp",[222,648,649],{"class":276}," \u002Ftmp\u002Fplain.yaml",[222,651,652],{"class":276}," \u002Ftmp\u002Fplain.yaml.sops.yaml\n",[222,654,655,657,659,661,663,666,669,672,675,677],{"class":224,"line":411},[222,656,602],{"class":601},[222,658,606],{"class":605},[222,660,609],{"class":276},[222,662,612],{"class":446},[222,664,665],{"class":456}," --config",[222,667,668],{"class":276}," .sops.yaml",[222,670,671],{"class":456}," -e",[222,673,674],{"class":276}," \u002Ftmp\u002Fplain.yaml.sops.yaml",[222,676,621],{"class":605},[222,678,679],{"class":276}," file.sops.yaml\n",[10,681,682],{},"Annoying but reliable. Documented it once, never had to figure it out again.",[75,684,686],{"id":685},"homepage-widgets-secret-management","Homepage Widgets & Secret Management",[10,688,689],{},"Once you understand the rules, Homepage is great:",[25,691,692,699,706],{},[28,693,694,695,698],{},"Secret keys must be prefixed ",[60,696,697],{},"HOMEPAGE_VAR_"," for template substitution",[28,700,701,702,705],{},"Use internal service URLs (",[60,703,704],{},"http:\u002F\u002Fservice.namespace.svc.cluster.local",")",[28,707,708,711],{},[60,709,710],{},"subPath"," mounts don't hot-reload — Reloader handles this automatically",[10,713,714,717],{},[31,715,716],{},"Gotcha: Complete secret updates"," — When updating Homepage secrets, don't patch individual fields. Delete and recreate the secret to ensure all credentials are present:",[186,719,721],{"className":437,"code":720,"language":439,"meta":194,"style":194},"kubectl delete secret -n homepage homepage-secrets\nkubectl apply -f new-secret.yaml  # With ALL credentials from SOPS\nkubectl rollout restart -n homepage deployment\u002Fhomepage\n",[60,722,723,741,757],{"__ignoreMap":194},[222,724,725,727,730,733,735,738],{"class":224,"line":225},[222,726,447],{"class":446},[222,728,729],{"class":276}," delete",[222,731,732],{"class":276}," secret",[222,734,457],{"class":456},[222,736,737],{"class":276}," homepage",[222,739,740],{"class":276}," homepage-secrets\n",[222,742,743,745,748,751,754],{"class":224,"line":236},[222,744,447],{"class":446},[222,746,747],{"class":276}," apply",[222,749,750],{"class":456}," -f",[222,752,753],{"class":276}," new-secret.yaml",[222,755,756],{"class":257},"  # With ALL credentials from SOPS\n",[222,758,759,761,764,767,769,771],{"class":224,"line":244},[222,760,447],{"class":446},[222,762,763],{"class":276}," rollout",[222,765,766],{"class":276}," restart",[222,768,457],{"class":456},[222,770,737],{"class":276},[222,772,773],{"class":276}," deployment\u002Fhomepage\n",[10,775,776],{},"This prevents the \"missing credentials\" issue where widgets show 401 errors because only some secrets were updated.",[10,778,779],{},"Widgets for Immich, Grafana, Traefik, CrowdSec, Plex, and Synology all working.",[75,781,783],{"id":782},"flux-reconciliation","Flux Reconciliation",[10,785,786],{},"The workflow is muscle memory now:",[186,788,790],{"className":437,"code":789,"language":439,"meta":194,"style":194},"# Edit the repo\n# Commit and push\ngit add -A && git commit -m \"feat: add thing\" && git push\n\n# Reconcile\nflux reconcile kustomization apps --with-source\n\n# Verify\nkubectl get helmrelease -A\n",[60,791,792,797,802,842,848,853,870,874,879],{"__ignoreMap":194},[222,793,794],{"class":224,"line":225},[222,795,796],{"class":257},"# Edit the repo\n",[222,798,799],{"class":224,"line":236},[222,800,801],{"class":257},"# Commit and push\n",[222,803,804,807,810,813,816,819,822,825,829,832,835,837,839],{"class":224,"line":244},[222,805,806],{"class":446},"git",[222,808,809],{"class":276}," add",[222,811,812],{"class":456}," -A",[222,814,815],{"class":232}," &&",[222,817,818],{"class":446}," git",[222,820,821],{"class":276}," commit",[222,823,824],{"class":456}," -m",[222,826,828],{"class":827},"siCPE"," \"",[222,830,831],{"class":276},"feat: add thing",[222,833,834],{"class":827},"\"",[222,836,815],{"class":232},[222,838,818],{"class":446},[222,840,841],{"class":276}," push\n",[222,843,844],{"class":224,"line":261},[222,845,847],{"emptyLinePlaceholder":846},true,"\n",[222,849,850],{"class":224,"line":269},[222,851,852],{"class":257},"# Reconcile\n",[222,854,855,858,861,864,867],{"class":224,"line":283},[222,856,857],{"class":446},"flux",[222,859,860],{"class":276}," reconcile",[222,862,863],{"class":276}," kustomization",[222,865,866],{"class":276}," apps",[222,868,869],{"class":456}," --with-source\n",[222,871,872],{"class":224,"line":411},[222,873,847],{"emptyLinePlaceholder":846},[222,875,876],{"class":224,"line":421},[222,877,878],{"class":257},"# Verify\n",[222,880,881,883,885,888],{"class":224,"line":550},[222,882,447],{"class":446},[222,884,450],{"class":276},[222,886,887],{"class":276}," helmrelease",[222,889,890],{"class":456}," -A\n",[75,892,894],{"id":893},"flux-web-ui","Flux Web UI",[10,896,897,898,901],{},"Flux has a built-in web UI (provided by the flux-operator) that shows the status of all GitOps resources at a glance. It's accessible at ",[60,899,900],{},"flux.example.com"," and provides:",[25,903,904,910,916,922],{},[28,905,906,909],{},[31,907,908],{},"Reconciliation status"," — Visual indicators for each Kustomization and HelmRelease",[28,911,912,915],{},[31,913,914],{},"Resource details"," — Click into any object to see its spec and status",[28,917,918,921],{},[31,919,920],{},"Error visibility"," — Quickly spot what's failing and why",[28,923,924,927],{},[31,925,926],{},"History & events"," — See recent reconciliation attempts and outcomes",[10,929,930,933,934,937],{},[31,931,932],{},"The enabling change:"," Add these values to your ",[60,935,936],{},"flux-instance"," HelmRelease:",[186,939,941],{"className":216,"code":940,"language":218,"meta":194,"style":194},"spec:\n  values:\n    instance:\n      web:\n        enabled: true\n        domain: flux.example.com\n",[60,942,943,949,956,963,970,979],{"__ignoreMap":194},[222,944,945,947],{"class":224,"line":225},[222,946,229],{"class":228},[222,948,233],{"class":232},[222,950,951,954],{"class":224,"line":236},[222,952,953],{"class":228},"  values",[222,955,233],{"class":232},[222,957,958,961],{"class":224,"line":244},[222,959,960],{"class":228},"    instance",[222,962,233],{"class":232},[222,964,965,968],{"class":224,"line":261},[222,966,967],{"class":228},"      web",[222,969,233],{"class":232},[222,971,972,975,977],{"class":224,"line":269},[222,973,974],{"class":228},"        enabled",[222,976,250],{"class":232},[222,978,291],{"class":253},[222,980,981,984,986],{"class":224,"line":283},[222,982,983],{"class":228},"        domain",[222,985,250],{"class":232},[222,987,988],{"class":276}," flux.example.com\n",[10,990,991,992,995,996,999],{},"This deploys the flux-operator UI service in the ",[60,993,994],{},"flux-system"," namespace. The service exposes port ",[60,997,998],{},"http-web"," (typically 8080) and is then exposed externally via Traefik IngressRoute:",[186,1001,1003],{"className":216,"code":1002,"language":218,"meta":194,"style":194},"apiVersion: traefik.io\u002Fv1alpha1\nkind: IngressRoute\nmetadata:\n  name: flux-ui\n  namespace: flux-system\nspec:\n  entryPoints:\n    - websecure\n  routes:\n    - kind: Rule\n      match: \"Host(`flux.example.com`)\"\n      services:\n        - name: flux-operator\n          port: http-web\n",[60,1004,1005,1013,1022,1028,1037,1046,1052,1059,1067,1074,1086,1101,1109,1123],{"__ignoreMap":194},[222,1006,1007,1009,1011],{"class":224,"line":225},[222,1008,360],{"class":228},[222,1010,250],{"class":232},[222,1012,365],{"class":276},[222,1014,1015,1017,1019],{"class":224,"line":236},[222,1016,370],{"class":228},[222,1018,250],{"class":232},[222,1020,1021],{"class":276}," IngressRoute\n",[222,1023,1024,1026],{"class":224,"line":244},[222,1025,380],{"class":228},[222,1027,233],{"class":232},[222,1029,1030,1032,1034],{"class":224,"line":261},[222,1031,387],{"class":228},[222,1033,250],{"class":232},[222,1035,1036],{"class":276}," flux-ui\n",[222,1038,1039,1041,1043],{"class":224,"line":269},[222,1040,397],{"class":228},[222,1042,250],{"class":232},[222,1044,1045],{"class":276}," flux-system\n",[222,1047,1048,1050],{"class":224,"line":283},[222,1049,229],{"class":228},[222,1051,233],{"class":232},[222,1053,1054,1057],{"class":224,"line":411},[222,1055,1056],{"class":228},"  entryPoints",[222,1058,233],{"class":232},[222,1060,1061,1064],{"class":224,"line":421},[222,1062,1063],{"class":232},"    -",[222,1065,1066],{"class":276}," websecure\n",[222,1068,1069,1072],{"class":224,"line":550},[222,1070,1071],{"class":228},"  routes",[222,1073,233],{"class":232},[222,1075,1076,1078,1081,1083],{"class":224,"line":556},[222,1077,1063],{"class":232},[222,1079,1080],{"class":228}," kind",[222,1082,250],{"class":232},[222,1084,1085],{"class":276}," Rule\n",[222,1087,1088,1091,1093,1095,1098],{"class":224,"line":562},[222,1089,1090],{"class":228},"      match",[222,1092,250],{"class":232},[222,1094,828],{"class":827},[222,1096,1097],{"class":276},"Host(`flux.example.com`)",[222,1099,1100],{"class":827},"\"\n",[222,1102,1104,1107],{"class":224,"line":1103},12,[222,1105,1106],{"class":228},"      services",[222,1108,233],{"class":232},[222,1110,1112,1115,1118,1120],{"class":224,"line":1111},13,[222,1113,1114],{"class":232},"        -",[222,1116,1117],{"class":228}," name",[222,1119,250],{"class":232},[222,1121,1122],{"class":276}," flux-operator\n",[222,1124,1126,1129,1131],{"class":224,"line":1125},14,[222,1127,1128],{"class":228},"          port",[222,1130,250],{"class":232},[222,1132,1133],{"class":276}," http-web\n",[10,1135,1136],{},"The UI is particularly helpful when debugging stuck reconciliations or understanding why a HelmRelease isn't deploying. It complements the CLI workflow for day-to-day operations.",[10,1138,1139,1142,1143,1145],{},[31,1140,1141],{},"Note:"," In the anonymized config, this is exposed via Traefik IngressRoute to ",[60,1144,900],{}," with CrowdSec and VoidAuth protection.",[75,1147,1149],{"id":1148},"custom-grafana-dashboards","Custom Grafana Dashboards",[10,1151,1152],{},"Four custom dashboards were built to make monitoring actionable:",[25,1154,1155,1161,1167,1173],{},[28,1156,1157,1160],{},[31,1158,1159],{},"Homelab Overview"," — Single-pane view of cluster health: CPU\u002Fmemory gauges for control plane and worker nodes, disk usage, and key service status",[28,1162,1163,1166],{},[31,1164,1165],{},"K8s Cluster Health"," — Deep dive into Kubernetes resource health, pod counts, and namespace-level metrics",[28,1168,1169,1172],{},[31,1170,1171],{},"Traefik Site Health"," — Request rates, response times, error rates, and service-level metrics for all ingress traffic",[28,1174,1175,1178],{},[31,1176,1177],{},"Immich Deep Dive"," — Photo library metrics: upload rates, storage usage, transcoding performance, and user activity",[10,1180,1181,1182,1185,1186,1189,1190,310],{},"Each dashboard is deployed as a ConfigMap with ",[60,1183,1184],{},"grafana_dashboard: \"1\""," label, auto-discovered by Grafana. They're maintained in ",[60,1187,1188],{},"infrastructure\u002Fconfigs\u002Fgrafana-dashboards.yaml"," and ",[60,1191,1192],{},"traefik-dashboard.yaml",[10,1194,1195,1198,1199,1203],{},[31,1196,1197],{},"Why custom dashboards?"," Standard Kubernetes dashboards show raw metrics; custom dashboards show ",[1200,1201,1202],"em",{},"your"," services in context. When Immich is slow, you can immediately see if it's storage, transcoding, or database queries.",[10,1205,1206,1209,1210,1217],{},[31,1207,1208],{},"More details:"," See the ",[1211,1212,1216],"a",{"href":1213,"rel":1214},"https:\u002F\u002Fgithub.com\u002Fmarr\u002Fflux-homelab-skill\u002Fblob\u002Fmain\u002Freferences\u002Fgrafana-dashboards.md",[1215],"nofollow","flux-homelab-skill Grafana dashboards reference"," for deployment patterns and maintenance.",[20,1219,1221],{"id":1220},"documentation-that-compounds","Documentation That Compounds",[10,1223,1224,1225,1228],{},"The most valuable output of this project isn't the cluster — it's the ",[60,1226,1227],{},"docs\u002Fsolutions\u002F"," directory in your infrastructure repository.",[10,1230,1231,1232,1235],{},"Every time I solved a non-trivial problem, I documented it: the symptoms, what didn't work, the fix, and why it works. This is the ",[31,1233,1234],{},"ce:compound"," pattern — each solution documented makes the next one faster.",[10,1237,1238,1239,1242,1243,1248],{},"The ",[60,1240,1241],{},"flux-homelab"," skill (available at ",[1211,1244,1247],{"href":1245,"rel":1246},"https:\u002F\u002Fgithub.com\u002Fmarr\u002Fflux-homelab-skill",[1215],"github.com\u002Fmarr\u002Fflux-homelab-skill",") encapsulates the patterns from this homelab — Flux reconciliation, SOPS secrets, Homepage widgets, and service-specific gotchas. It's like a runbook that travels with the agent.",[10,1250,1251,1252,1255],{},"When I ask \"why is Immich down,\" it knows to check storage class first, then node scheduling, then Flux kustomization health. It remembers that node names changed, that OCI chart sources have two valid patterns, and that CrowdSec blocks ",[60,1253,1254],{},"wget"," user agents.",[10,1257,1258],{},[31,1259,1260],{},"Skill contents include:",[25,1262,1263,1269,1275,1288],{},[28,1264,1265,1268],{},[31,1266,1267],{},"Cluster hygiene"," — Detecting mixed Endpoints + EndpointSlice issues, orphaned services, hardcoded LAN IPs",[28,1270,1271,1274],{},[31,1272,1273],{},"Flux reconciliation"," — Proper sequence for GitRepository → Kustomization → HelmRelease",[28,1276,1277,1280,1281,306,1284,1287],{},[31,1278,1279],{},"HelmRelease patterns"," — ",[60,1282,1283],{},"upgrade.force",[60,1285,1286],{},"serverSideApply"," interactions, stuck kustomization recovery",[28,1289,1290,1293],{},[31,1291,1292],{},"Secret management"," — SOPS file structure, secret synchronization, validation workflows",[10,1295,1238,1296,1298,1299,1304],{},[31,1297,1234],{}," pattern (referenced from ",[1211,1300,1303],{"href":1301,"rel":1302},"https:\u002F\u002Fevery.to\u002Fguides\u002Fcompound-engineering",[1215],"Compound Engineering",") is a structured approach to documentation — researching problems, assembling solutions, writing them down, and validating that they work. Each documented solution makes the next one faster. This pattern applies to any project, not just homelabs.",[20,1306,1308],{"id":1307},"getting-started-with-flux-homelab-experimentation","Getting Started with Flux Homelab Experimentation",[10,1310,1311],{},"If you want to try this yourself:",[75,1313,1315],{"id":1314},"_1-pick-your-platform","1. Pick Your Platform",[10,1317,1318,1321,1322,1325],{},[31,1319,1320],{},"Talos"," is great if you want an immutable, API-driven cluster. ",[31,1323,1324],{},"k3s"," is great if you want something more familiar. Either way, you need:",[25,1327,1328,1331,1334],{},[28,1329,1330],{},"A machine (old laptop, NUC, Mac Mini, VM)",[28,1332,1333],{},"A git repo for your Flux configuration",[28,1335,1336],{},"Time to break things",[75,1338,1340],{"id":1339},"_2-bootstrap-flux","2. Bootstrap Flux",[10,1342,1343,1344,1349],{},"Follow the ",[1211,1345,1348],{"href":1346,"rel":1347},"https:\u002F\u002Ffluxcd.io\u002Fflux\u002Fget-started\u002F",[1215],"official Flux getting started guide"," to install Flux on your cluster, then bootstrap your GitOps repository:",[186,1351,1353],{"className":437,"code":1352,"language":439,"meta":194,"style":194},"flux bootstrap github \\\n  --owner=your-username \\\n  --repository=homelab-infra \\\n  --branch=main \\\n  --path=clusters\u002Fmy-cluster\n",[60,1354,1355,1369,1376,1383,1390],{"__ignoreMap":194},[222,1356,1357,1359,1362,1365],{"class":224,"line":225},[222,1358,857],{"class":446},[222,1360,1361],{"class":276}," bootstrap",[222,1363,1364],{"class":276}," github",[222,1366,1368],{"class":1367},"sQeA1"," \\\n",[222,1370,1371,1374],{"class":224,"line":236},[222,1372,1373],{"class":456},"  --owner=your-username",[222,1375,1368],{"class":1367},[222,1377,1378,1381],{"class":224,"line":244},[222,1379,1380],{"class":456},"  --repository=homelab-infra",[222,1382,1368],{"class":1367},[222,1384,1385,1388],{"class":224,"line":261},[222,1386,1387],{"class":456},"  --branch=main",[222,1389,1368],{"class":1367},[222,1391,1392],{"class":224,"line":269},[222,1393,1394],{"class":456},"  --path=clusters\u002Fmy-cluster\n",[10,1396,1397,1398,1189,1401,1404],{},"This creates the bootstrap kustomization that points to your ",[60,1399,1400],{},"infrastructure\u002F",[60,1402,1403],{},"apps\u002F"," directories.",[10,1406,1407],{},[31,1408,1409],{},"Flux v2 Gotchas:",[25,1411,1412,1424,1434,1447,1453,1478,1539],{},[28,1413,1414,1417,1418,1420,1421,1423],{},[31,1415,1416],{},"Namespace changes"," — In v2, Flux components run in ",[60,1419,994],{}," by default. Some older guides reference ",[60,1422,857],{}," namespace.",[28,1425,1426,1429,1430,1433],{},[31,1427,1428],{},"Kustomize controller"," — The controller now handles kustomizations natively; you don't need separate ",[60,1431,1432],{},"kustomize"," CLI workflows.",[28,1435,1436,1439,1440,1189,1443,1446],{},[31,1437,1438],{},"GitHub token scope"," — The bootstrap command needs a token with ",[60,1441,1442],{},"repo",[60,1444,1445],{},"workflow"," permissions. Personal Access Tokens work, but GitHub Apps are recommended for production.",[28,1448,1449,1452],{},[31,1450,1451],{},"Path separators"," — Use forward slashes even on Windows; Flux paths are Git repository paths, not filesystem paths.",[28,1454,1455,1458,1459,1462,1463,1466,1467,346,1470,1473,1474,1477],{},[31,1456,1457],{},"OCI chart sources"," — Flux 2.8+ uses ",[60,1460,1461],{},"source.toolkit.fluxcd.io\u002Fv1"," for GitRepository\u002FHelmRepository\u002FOCIRepository, but ",[60,1464,1465],{},"helm.toolkit.fluxcd.io\u002Fv2"," for HelmRelease. OCI-backed charts have two valid patterns: ",[60,1468,1469],{},"HelmRepository",[60,1471,1472],{},"type: oci",", or ",[60,1475,1476],{},"OCIRepository"," directly. If one pattern fails, try the other.",[28,1479,1480,1483,1484,1486,1487,209,1489,1491,1492],{},[31,1481,1482],{},"HelmRelease serverSideApply"," — The ",[60,1485,200],{}," field is a boolean, but ",[60,1488,208],{},[60,1490,212],{},"). Mixing them incorrectly causes validation errors. Use:\n",[186,1493,1495],{"className":216,"code":1494,"language":218,"meta":194,"style":194},"install:\n  serverSideApply: false\nupgrade:\n  serverSideApply: disabled\n  force: true\n",[60,1496,1497,1504,1514,1521,1530],{"__ignoreMap":194},[222,1498,1499,1502],{"class":224,"line":225},[222,1500,1501],{"class":228},"install",[222,1503,233],{"class":232},[222,1505,1506,1509,1511],{"class":224,"line":236},[222,1507,1508],{"class":228},"  serverSideApply",[222,1510,250],{"class":232},[222,1512,1513],{"class":253}," false\n",[222,1515,1516,1519],{"class":224,"line":244},[222,1517,1518],{"class":228},"upgrade",[222,1520,233],{"class":232},[222,1522,1523,1525,1527],{"class":224,"line":261},[222,1524,1508],{"class":228},[222,1526,250],{"class":232},[222,1528,1529],{"class":276}," disabled\n",[222,1531,1532,1535,1537],{"class":224,"line":269},[222,1533,1534],{"class":228},"  force",[222,1536,250],{"class":232},[222,1538,291],{"class":253},[28,1540,1541,1544,1545,1548,1549,1552,1553,1556,1557,1560,1561,1564],{},[31,1542,1543],{},"Stuck kustomizations"," — If a ",[60,1546,1547],{},"HelmRelease"," health check fails, downstream kustomizations with ",[60,1550,1551],{},"spec.dependsOn"," will wait indefinitely. Check ",[60,1554,1555],{},"kubectl describe kustomization \u003Cname> -n flux-system"," for ",[60,1558,1559],{},"HealthCheckFailed",". High ",[60,1562,1563],{},"helm-controller"," CPU often indicates a failing Helm release in a retry loop.",[75,1566,1568],{"id":1567},"_3-structure-your-repo","3. Structure Your Repo",[186,1570,1573],{"className":1571,"code":1572,"language":191},[189],"homelab-infra\u002F\n├── clusters\u002F           # Flux bootstrap\n├── infrastructure\u002F\n│   ├── controllers\u002F    # Traefik, Prometheus, etc.\n│   └── configs\u002F        # Routes, monitoring, alerts\n└── apps\u002F\n    └── my-app\u002F         # namespace, kustomization, helmrelease\n",[60,1574,1572],{"__ignoreMap":194},[75,1576,1578],{"id":1577},"_4-add-sops-early","4. Add SOPS Early",[10,1580,1581],{},"Set up age encryption before you have any secrets:",[186,1583,1585],{"className":437,"code":1584,"language":439,"meta":194,"style":194},"age-keygen -o ~\u002F.config\u002Fsops\u002Fage\u002Fkeys.txt\n# Add the public key to .sops.yaml\n",[60,1586,1587,1598],{"__ignoreMap":194},[222,1588,1589,1592,1595],{"class":224,"line":225},[222,1590,1591],{"class":446},"age-keygen",[222,1593,1594],{"class":456}," -o",[222,1596,1597],{"class":276}," ~\u002F.config\u002Fsops\u002Fage\u002Fkeys.txt\n",[222,1599,1600],{"class":224,"line":236},[222,1601,1602],{"class":257},"# Add the public key to .sops.yaml\n",[75,1604,1606],{"id":1605},"_5-build-incrementally","5. Build Incrementally",[10,1608,1609],{},"Don't try to deploy everything at once. Start with:",[158,1611,1612,1615,1618,1621,1624],{},[28,1613,1614],{},"Traefik (ingress)",[28,1616,1617],{},"One app (immich, nextcloud, whatever)",[28,1619,1620],{},"Homepage (dashboard)",[28,1622,1623],{},"Monitoring (Prometheus + Grafana)",[28,1625,1626],{},"Security (CrowdSec + auth)",[10,1628,1629],{},"Each step teaches you something. Document each step.",[75,1631,1633],{"id":1632},"_6-document-as-you-go","6. Document as You Go",[10,1635,1636,1637,1639],{},"Create a ",[60,1638,1227],{}," directory in your repo. When you solve something non-trivial, write it down. Use YAML frontmatter so future-you (or your AI assistant) can search by module, tags, and problem type.",[10,1641,1642],{},"The compounding effect is real. Week 1 I spent hours debugging Loki crash loops. Week 12, I checked the documented solution and fixed a similar issue in minutes.",[20,1644,1646],{"id":1645},"whats-next","What's Next",[10,1648,1649],{},"The cluster keeps growing. Next up:",[25,1651,1652,1655,1658,1661],{},[28,1653,1654],{},"Gitea Actions for CI (self-hosted runners in-cluster)",[28,1656,1657],{},"Synology Drive integration (finally got the Traefik routing right)",[28,1659,1660],{},"Better alerting rules in Alertmanager",[28,1662,1663],{},"Maybe a second node for actual HA",[10,1665,1666],{},"The homelab isn't a project with an end state. It's an ongoing experiment where each iteration teaches something new. The key is capturing those lessons so they compound.",[1668,1669],"hr",{},[10,1671,1672],{},[1200,1673,1674,1675,310],{},"For more on AI-assisted development, see my post on ",[1211,1676,1678],{"href":1677},"\u002Fblog\u002Fknowledge-graph-obsidian-mcp","Building a Knowledge Graph with Obsidian and MCP",[1680,1681,1682],"style",{},"html pre.shiki code .sR7ES, html code.shiki .sR7ES{--shiki-light:#E2931D;--shiki-default:#6F42C1;--shiki-dark:#B392F0;--shiki-sepia:#A6E22E}html pre.shiki code .sLACW, html code.shiki .sLACW{--shiki-light:#91B859;--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sFhLe, html code.shiki .sFhLe{--shiki-light:#91B859;--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .ss7Ak, html code.shiki .ss7Ak{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#6A737D;--shiki-default-font-style:inherit;--shiki-dark:#6A737D;--shiki-dark-font-style:inherit;--shiki-sepia:#88846F;--shiki-sepia-font-style:inherit}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html .sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html.sepia .shiki span {color: var(--shiki-sepia);background: var(--shiki-sepia-bg);font-style: var(--shiki-sepia-font-style);font-weight: var(--shiki-sepia-font-weight);text-decoration: var(--shiki-sepia-text-decoration);}html pre.shiki code .ss--_, html code.shiki .ss--_{--shiki-light:#90A4AE;--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .sGXK2, html code.shiki .sGXK2{--shiki-light:#39ADB5;--shiki-default:#D73A49;--shiki-dark:#F97583;--shiki-sepia:#F92672}html pre.shiki code .swvn1, html code.shiki .swvn1{--shiki-light:#39ADB5;--shiki-default:#24292E;--shiki-dark:#E1E4E8;--shiki-sepia:#F8F8F2}html pre.shiki code .siCPE, html code.shiki .siCPE{--shiki-light:#39ADB5;--shiki-default:#032F62;--shiki-dark:#9ECBFF;--shiki-sepia:#E6DB74}html pre.shiki code .sQeA1, html code.shiki .sQeA1{--shiki-light:#90A4AE;--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sHsBP, html code.shiki .sHsBP{--shiki-light:#E53935;--shiki-default:#22863A;--shiki-dark:#85E89D;--shiki-sepia:#F92672}html pre.shiki code .s8HiA, html code.shiki .s8HiA{--shiki-light:#FF5370;--shiki-default:#005CC5;--shiki-dark:#79B8FF;--shiki-sepia:#AE81FF}html pre.shiki code .sRxSC, html code.shiki .sRxSC{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit;--shiki-sepia:#F92672;--shiki-sepia-font-style:inherit}",{"title":194,"searchDepth":236,"depth":236,"links":1684},[1685,1686,1693,1700,1701,1709],{"id":22,"depth":236,"text":23},{"id":72,"depth":236,"text":73,"children":1687},[1688,1689,1690,1691,1692],{"id":77,"depth":244,"text":78},{"id":152,"depth":244,"text":153},{"id":180,"depth":244,"text":181},{"id":318,"depth":244,"text":319},{"id":476,"depth":244,"text":477},{"id":574,"depth":236,"text":575,"children":1694},[1695,1696,1697,1698,1699],{"id":578,"depth":244,"text":579},{"id":685,"depth":244,"text":686},{"id":782,"depth":244,"text":783},{"id":893,"depth":244,"text":894},{"id":1148,"depth":244,"text":1149},{"id":1220,"depth":236,"text":1221},{"id":1307,"depth":236,"text":1308,"children":1702},[1703,1704,1705,1706,1707,1708],{"id":1314,"depth":244,"text":1315},{"id":1339,"depth":244,"text":1340},{"id":1567,"depth":244,"text":1568},{"id":1577,"depth":244,"text":1578},{"id":1605,"depth":244,"text":1606},{"id":1632,"depth":244,"text":1633},{"id":1645,"depth":236,"text":1646},"2026-04-29","How I built a home Kubernetes cluster with Flux, SOPS secrets, and a dashboard that actually works — plus the skills that helped document everything along the way.","md",null,{},"\u002Fblog\u002Fhomelab-flux-gitops-journey",{"title":5,"description":1711},"blog\u002Fhomelab-flux-gitops-journey","jlO27QufsoCJ0VmPIV4tSA3X7BBufOK1CcKKVXTWhDE",{"id":4,"title":5,"body":1720,"date":1710,"description":1711,"extension":1712,"image":1713,"meta":2909,"navigation":846,"path":1715,"seo":2910,"stem":1717,"__hash__":1718},{"type":7,"value":1721,"toc":2882},[1722,1724,1726,1728,1730,1758,1760,1762,1768,1770,1804,1810,1812,1814,1824,1828,1830,1832,1837,1847,1897,1907,1911,1913,1915,1933,1997,2001,2021,2025,2027,2031,2037,2091,2095,2097,2099,2103,2173,2175,2177,2179,2193,2197,2241,2243,2245,2247,2249,2327,2329,2333,2351,2357,2401,2407,2519,2521,2527,2529,2531,2549,2557,2563,2570,2572,2576,2580,2587,2591,2595,2617,2624,2626,2628,2630,2636,2644,2646,2651,2687,2693,2697,2803,2805,2810,2812,2814,2830,2832,2834,2846,2848,2850,2854,2856,2858,2860,2870,2872,2874,2880],[10,1723,12],{},[10,1725,15],{},[10,1727,18],{},[20,1729,23],{"id":22},[25,1731,1732,1736,1740,1744,1748,1754],{},[28,1733,1734,34],{},[31,1735,33],{},[28,1737,1738,40],{},[31,1739,39],{},[28,1741,1742,46],{},[31,1743,45],{},[28,1745,1746,52],{},[31,1747,51],{},[28,1749,1750,58,1752,63],{},[31,1751,57],{},[60,1753,62],{},[28,1755,1756,69],{},[31,1757,68],{},[20,1759,73],{"id":72},[75,1761,78],{"id":77},[10,1763,81,1764,85,1766,89],{},[60,1765,84],{},[60,1767,88],{},[10,1769,92],{},[94,1771,1772,1782],{},[97,1773,1774],{},[100,1775,1776,1778,1780],{},[103,1777,105],{},[103,1779,108],{},[103,1781,111],{},[113,1783,1784,1794],{},[100,1785,1786,1790,1792],{},[118,1787,1788],{},[60,1789,122],{},[118,1791,125],{},[118,1793,128],{},[100,1795,1796,1800,1802],{},[118,1797,1798],{},[60,1799,84],{},[118,1801,137],{},[118,1803,140],{},[10,1805,1806,146,1808,149],{},[31,1807,145],{},[60,1809,122],{},[75,1811,153],{"id":152},[10,1813,156],{},[158,1815,1816,1818,1820,1822],{},[28,1817,162],{},[28,1819,165],{},[28,1821,168],{},[28,1823,171],{},[10,1825,1826,177],{},[31,1827,176],{},[75,1829,181],{"id":180},[10,1831,184],{},[186,1833,1835],{"className":1834,"code":190,"language":191},[189],[60,1836,190],{"__ignoreMap":194},[10,1838,197,1839,201,1841,205,1843,209,1845,213],{},[60,1840,200],{},[60,1842,204],{},[60,1844,208],{},[60,1846,212],{},[186,1848,1849],{"className":216,"code":217,"language":218,"meta":194,"style":194},[60,1850,1851,1857,1863,1873,1879,1889],{"__ignoreMap":194},[222,1852,1853,1855],{"class":224,"line":225},[222,1854,229],{"class":228},[222,1856,233],{"class":232},[222,1858,1859,1861],{"class":224,"line":236},[222,1860,239],{"class":228},[222,1862,233],{"class":232},[222,1864,1865,1867,1869,1871],{"class":224,"line":244},[222,1866,247],{"class":228},[222,1868,250],{"class":232},[222,1870,254],{"class":253},[222,1872,258],{"class":257},[222,1874,1875,1877],{"class":224,"line":261},[222,1876,264],{"class":228},[222,1878,233],{"class":232},[222,1880,1881,1883,1885,1887],{"class":224,"line":269},[222,1882,247],{"class":228},[222,1884,250],{"class":232},[222,1886,277],{"class":276},[222,1888,280],{"class":257},[222,1890,1891,1893,1895],{"class":224,"line":283},[222,1892,286],{"class":228},[222,1894,250],{"class":232},[222,1896,291],{"class":253},[10,1898,294,1899,298,1901,302,1903,306,1905,310],{},[60,1900,297],{},[60,1902,301],{},[60,1904,305],{},[60,1906,309],{},[10,1908,1909,315],{},[31,1910,145],{},[75,1912,319],{"id":318},[10,1914,322],{},[158,1916,1917,1921,1925],{},[28,1918,1919,330],{},[31,1920,329],{},[28,1922,1923,336],{},[31,1924,335],{},[28,1926,1927,342,1929,346,1931,350],{},[31,1928,341],{},[60,1930,345],{},[60,1932,349],{},[186,1934,1935],{"className":216,"code":353,"language":218,"meta":194,"style":194},[60,1936,1937,1945,1953,1959,1967,1975,1981,1989],{"__ignoreMap":194},[222,1938,1939,1941,1943],{"class":224,"line":225},[222,1940,360],{"class":228},[222,1942,250],{"class":232},[222,1944,365],{"class":276},[222,1946,1947,1949,1951],{"class":224,"line":236},[222,1948,370],{"class":228},[222,1950,250],{"class":232},[222,1952,375],{"class":276},[222,1954,1955,1957],{"class":224,"line":244},[222,1956,380],{"class":228},[222,1958,233],{"class":232},[222,1960,1961,1963,1965],{"class":224,"line":261},[222,1962,387],{"class":228},[222,1964,250],{"class":232},[222,1966,392],{"class":276},[222,1968,1969,1971,1973],{"class":224,"line":269},[222,1970,397],{"class":228},[222,1972,250],{"class":232},[222,1974,402],{"class":276},[222,1976,1977,1979],{"class":224,"line":283},[222,1978,229],{"class":228},[222,1980,233],{"class":232},[222,1982,1983,1985,1987],{"class":224,"line":411},[222,1984,414],{"class":228},[222,1986,250],{"class":232},[222,1988,291],{"class":253},[222,1990,1991,1993,1995],{"class":224,"line":421},[222,1992,424],{"class":228},[222,1994,250],{"class":232},[222,1996,291],{"class":253},[10,1998,1999,434],{},[31,2000,433],{},[186,2002,2003],{"className":437,"code":438,"language":439,"meta":194,"style":194},[60,2004,2005],{"__ignoreMap":194},[222,2006,2007,2009,2011,2013,2015,2017,2019],{"class":224,"line":225},[222,2008,447],{"class":446},[222,2010,450],{"class":276},[222,2012,453],{"class":276},[222,2014,457],{"class":456},[222,2016,460],{"class":276},[222,2018,463],{"class":276},[222,2020,466],{"class":257},[10,2022,469,2023,473],{},[60,2024,472],{},[75,2026,477],{"id":476},[10,2028,480,2029,484],{},[60,2030,483],{},[10,2032,2033,490,2035,494],{},[31,2034,489],{},[60,2036,493],{},[186,2038,2039],{"className":216,"code":497,"language":218,"meta":194,"style":194},[60,2040,2041,2045,2051,2059,2063,2067,2071,2075,2079,2083,2087],{"__ignoreMap":194},[222,2042,2043],{"class":224,"line":225},[222,2044,504],{"class":257},[222,2046,2047,2049],{"class":224,"line":236},[222,2048,509],{"class":228},[222,2050,233],{"class":232},[222,2052,2053,2055,2057],{"class":224,"line":244},[222,2054,516],{"class":228},[222,2056,250],{"class":232},[222,2058,522],{"class":521},[222,2060,2061],{"class":224,"line":261},[222,2062,527],{"class":276},[222,2064,2065],{"class":224,"line":269},[222,2066,532],{"class":276},[222,2068,2069],{"class":224,"line":283},[222,2070,537],{"class":276},[222,2072,2073],{"class":224,"line":411},[222,2074,542],{"class":276},[222,2076,2077],{"class":224,"line":421},[222,2078,547],{"class":276},[222,2080,2081],{"class":224,"line":550},[222,2082,553],{"class":276},[222,2084,2085],{"class":224,"line":556},[222,2086,559],{"class":276},[222,2088,2089],{"class":224,"line":562},[222,2090,565],{"class":276},[10,2092,2093,571],{},[31,2094,570],{},[20,2096,575],{"id":574},[75,2098,579],{"id":578},[10,2100,582,2101,586],{},[60,2102,585],{},[186,2104,2105],{"className":437,"code":589,"language":439,"meta":194,"style":194},[60,2106,2107,2111,2129,2133,2139,2143,2151],{"__ignoreMap":194},[222,2108,2109],{"class":224,"line":225},[222,2110,596],{"class":257},[222,2112,2113,2115,2117,2119,2121,2123,2125,2127],{"class":224,"line":236},[222,2114,602],{"class":601},[222,2116,606],{"class":605},[222,2118,609],{"class":276},[222,2120,612],{"class":446},[222,2122,615],{"class":456},[222,2124,618],{"class":276},[222,2126,621],{"class":605},[222,2128,624],{"class":276},[222,2130,2131],{"class":224,"line":244},[222,2132,629],{"class":257},[222,2134,2135,2137],{"class":224,"line":261},[222,2136,634],{"class":446},[222,2138,624],{"class":276},[222,2140,2141],{"class":224,"line":269},[222,2142,641],{"class":257},[222,2144,2145,2147,2149],{"class":224,"line":283},[222,2146,646],{"class":446},[222,2148,649],{"class":276},[222,2150,652],{"class":276},[222,2152,2153,2155,2157,2159,2161,2163,2165,2167,2169,2171],{"class":224,"line":411},[222,2154,602],{"class":601},[222,2156,606],{"class":605},[222,2158,609],{"class":276},[222,2160,612],{"class":446},[222,2162,665],{"class":456},[222,2164,668],{"class":276},[222,2166,671],{"class":456},[222,2168,674],{"class":276},[222,2170,621],{"class":605},[222,2172,679],{"class":276},[10,2174,682],{},[75,2176,686],{"id":685},[10,2178,689],{},[25,2180,2181,2185,2189],{},[28,2182,694,2183,698],{},[60,2184,697],{},[28,2186,701,2187,705],{},[60,2188,704],{},[28,2190,2191,711],{},[60,2192,710],{},[10,2194,2195,717],{},[31,2196,716],{},[186,2198,2199],{"className":437,"code":720,"language":439,"meta":194,"style":194},[60,2200,2201,2215,2227],{"__ignoreMap":194},[222,2202,2203,2205,2207,2209,2211,2213],{"class":224,"line":225},[222,2204,447],{"class":446},[222,2206,729],{"class":276},[222,2208,732],{"class":276},[222,2210,457],{"class":456},[222,2212,737],{"class":276},[222,2214,740],{"class":276},[222,2216,2217,2219,2221,2223,2225],{"class":224,"line":236},[222,2218,447],{"class":446},[222,2220,747],{"class":276},[222,2222,750],{"class":456},[222,2224,753],{"class":276},[222,2226,756],{"class":257},[222,2228,2229,2231,2233,2235,2237,2239],{"class":224,"line":244},[222,2230,447],{"class":446},[222,2232,763],{"class":276},[222,2234,766],{"class":276},[222,2236,457],{"class":456},[222,2238,737],{"class":276},[222,2240,773],{"class":276},[10,2242,776],{},[10,2244,779],{},[75,2246,783],{"id":782},[10,2248,786],{},[186,2250,2251],{"className":437,"code":789,"language":439,"meta":194,"style":194},[60,2252,2253,2257,2261,2289,2293,2297,2309,2313,2317],{"__ignoreMap":194},[222,2254,2255],{"class":224,"line":225},[222,2256,796],{"class":257},[222,2258,2259],{"class":224,"line":236},[222,2260,801],{"class":257},[222,2262,2263,2265,2267,2269,2271,2273,2275,2277,2279,2281,2283,2285,2287],{"class":224,"line":244},[222,2264,806],{"class":446},[222,2266,809],{"class":276},[222,2268,812],{"class":456},[222,2270,815],{"class":232},[222,2272,818],{"class":446},[222,2274,821],{"class":276},[222,2276,824],{"class":456},[222,2278,828],{"class":827},[222,2280,831],{"class":276},[222,2282,834],{"class":827},[222,2284,815],{"class":232},[222,2286,818],{"class":446},[222,2288,841],{"class":276},[222,2290,2291],{"class":224,"line":261},[222,2292,847],{"emptyLinePlaceholder":846},[222,2294,2295],{"class":224,"line":269},[222,2296,852],{"class":257},[222,2298,2299,2301,2303,2305,2307],{"class":224,"line":283},[222,2300,857],{"class":446},[222,2302,860],{"class":276},[222,2304,863],{"class":276},[222,2306,866],{"class":276},[222,2308,869],{"class":456},[222,2310,2311],{"class":224,"line":411},[222,2312,847],{"emptyLinePlaceholder":846},[222,2314,2315],{"class":224,"line":421},[222,2316,878],{"class":257},[222,2318,2319,2321,2323,2325],{"class":224,"line":550},[222,2320,447],{"class":446},[222,2322,450],{"class":276},[222,2324,887],{"class":276},[222,2326,890],{"class":456},[75,2328,894],{"id":893},[10,2330,897,2331,901],{},[60,2332,900],{},[25,2334,2335,2339,2343,2347],{},[28,2336,2337,909],{},[31,2338,908],{},[28,2340,2341,915],{},[31,2342,914],{},[28,2344,2345,921],{},[31,2346,920],{},[28,2348,2349,927],{},[31,2350,926],{},[10,2352,2353,933,2355,937],{},[31,2354,932],{},[60,2356,936],{},[186,2358,2359],{"className":216,"code":940,"language":218,"meta":194,"style":194},[60,2360,2361,2367,2373,2379,2385,2393],{"__ignoreMap":194},[222,2362,2363,2365],{"class":224,"line":225},[222,2364,229],{"class":228},[222,2366,233],{"class":232},[222,2368,2369,2371],{"class":224,"line":236},[222,2370,953],{"class":228},[222,2372,233],{"class":232},[222,2374,2375,2377],{"class":224,"line":244},[222,2376,960],{"class":228},[222,2378,233],{"class":232},[222,2380,2381,2383],{"class":224,"line":261},[222,2382,967],{"class":228},[222,2384,233],{"class":232},[222,2386,2387,2389,2391],{"class":224,"line":269},[222,2388,974],{"class":228},[222,2390,250],{"class":232},[222,2392,291],{"class":253},[222,2394,2395,2397,2399],{"class":224,"line":283},[222,2396,983],{"class":228},[222,2398,250],{"class":232},[222,2400,988],{"class":276},[10,2402,991,2403,995,2405,999],{},[60,2404,994],{},[60,2406,998],{},[186,2408,2409],{"className":216,"code":1002,"language":218,"meta":194,"style":194},[60,2410,2411,2419,2427,2433,2441,2449,2455,2461,2467,2473,2483,2495,2501,2511],{"__ignoreMap":194},[222,2412,2413,2415,2417],{"class":224,"line":225},[222,2414,360],{"class":228},[222,2416,250],{"class":232},[222,2418,365],{"class":276},[222,2420,2421,2423,2425],{"class":224,"line":236},[222,2422,370],{"class":228},[222,2424,250],{"class":232},[222,2426,1021],{"class":276},[222,2428,2429,2431],{"class":224,"line":244},[222,2430,380],{"class":228},[222,2432,233],{"class":232},[222,2434,2435,2437,2439],{"class":224,"line":261},[222,2436,387],{"class":228},[222,2438,250],{"class":232},[222,2440,1036],{"class":276},[222,2442,2443,2445,2447],{"class":224,"line":269},[222,2444,397],{"class":228},[222,2446,250],{"class":232},[222,2448,1045],{"class":276},[222,2450,2451,2453],{"class":224,"line":283},[222,2452,229],{"class":228},[222,2454,233],{"class":232},[222,2456,2457,2459],{"class":224,"line":411},[222,2458,1056],{"class":228},[222,2460,233],{"class":232},[222,2462,2463,2465],{"class":224,"line":421},[222,2464,1063],{"class":232},[222,2466,1066],{"class":276},[222,2468,2469,2471],{"class":224,"line":550},[222,2470,1071],{"class":228},[222,2472,233],{"class":232},[222,2474,2475,2477,2479,2481],{"class":224,"line":556},[222,2476,1063],{"class":232},[222,2478,1080],{"class":228},[222,2480,250],{"class":232},[222,2482,1085],{"class":276},[222,2484,2485,2487,2489,2491,2493],{"class":224,"line":562},[222,2486,1090],{"class":228},[222,2488,250],{"class":232},[222,2490,828],{"class":827},[222,2492,1097],{"class":276},[222,2494,1100],{"class":827},[222,2496,2497,2499],{"class":224,"line":1103},[222,2498,1106],{"class":228},[222,2500,233],{"class":232},[222,2502,2503,2505,2507,2509],{"class":224,"line":1111},[222,2504,1114],{"class":232},[222,2506,1117],{"class":228},[222,2508,250],{"class":232},[222,2510,1122],{"class":276},[222,2512,2513,2515,2517],{"class":224,"line":1125},[222,2514,1128],{"class":228},[222,2516,250],{"class":232},[222,2518,1133],{"class":276},[10,2520,1136],{},[10,2522,2523,1142,2525,1145],{},[31,2524,1141],{},[60,2526,900],{},[75,2528,1149],{"id":1148},[10,2530,1152],{},[25,2532,2533,2537,2541,2545],{},[28,2534,2535,1160],{},[31,2536,1159],{},[28,2538,2539,1166],{},[31,2540,1165],{},[28,2542,2543,1172],{},[31,2544,1171],{},[28,2546,2547,1178],{},[31,2548,1177],{},[10,2550,1181,2551,1185,2553,1189,2555,310],{},[60,2552,1184],{},[60,2554,1188],{},[60,2556,1192],{},[10,2558,2559,1198,2561,1203],{},[31,2560,1197],{},[1200,2562,1202],{},[10,2564,2565,1209,2567,1217],{},[31,2566,1208],{},[1211,2568,1216],{"href":1213,"rel":2569},[1215],[20,2571,1221],{"id":1220},[10,2573,1224,2574,1228],{},[60,2575,1227],{},[10,2577,1231,2578,1235],{},[31,2579,1234],{},[10,2581,1238,2582,1242,2584,1248],{},[60,2583,1241],{},[1211,2585,1247],{"href":1245,"rel":2586},[1215],[10,2588,1251,2589,1255],{},[60,2590,1254],{},[10,2592,2593],{},[31,2594,1260],{},[25,2596,2597,2601,2605,2613],{},[28,2598,2599,1268],{},[31,2600,1267],{},[28,2602,2603,1274],{},[31,2604,1273],{},[28,2606,2607,1280,2609,306,2611,1287],{},[31,2608,1279],{},[60,2610,1283],{},[60,2612,1286],{},[28,2614,2615,1293],{},[31,2616,1292],{},[10,2618,1238,2619,1298,2621,1304],{},[31,2620,1234],{},[1211,2622,1303],{"href":1301,"rel":2623},[1215],[20,2625,1308],{"id":1307},[10,2627,1311],{},[75,2629,1315],{"id":1314},[10,2631,2632,1321,2634,1325],{},[31,2633,1320],{},[31,2635,1324],{},[25,2637,2638,2640,2642],{},[28,2639,1330],{},[28,2641,1333],{},[28,2643,1336],{},[75,2645,1340],{"id":1339},[10,2647,1343,2648,1349],{},[1211,2649,1348],{"href":1346,"rel":2650},[1215],[186,2652,2653],{"className":437,"code":1352,"language":439,"meta":194,"style":194},[60,2654,2655,2665,2671,2677,2683],{"__ignoreMap":194},[222,2656,2657,2659,2661,2663],{"class":224,"line":225},[222,2658,857],{"class":446},[222,2660,1361],{"class":276},[222,2662,1364],{"class":276},[222,2664,1368],{"class":1367},[222,2666,2667,2669],{"class":224,"line":236},[222,2668,1373],{"class":456},[222,2670,1368],{"class":1367},[222,2672,2673,2675],{"class":224,"line":244},[222,2674,1380],{"class":456},[222,2676,1368],{"class":1367},[222,2678,2679,2681],{"class":224,"line":261},[222,2680,1387],{"class":456},[222,2682,1368],{"class":1367},[222,2684,2685],{"class":224,"line":269},[222,2686,1394],{"class":456},[10,2688,1397,2689,1189,2691,1404],{},[60,2690,1400],{},[60,2692,1403],{},[10,2694,2695],{},[31,2696,1409],{},[25,2698,2699,2707,2713,2721,2725,2739,2789],{},[28,2700,2701,1417,2703,1420,2705,1423],{},[31,2702,1416],{},[60,2704,994],{},[60,2706,857],{},[28,2708,2709,1429,2711,1433],{},[31,2710,1428],{},[60,2712,1432],{},[28,2714,2715,1439,2717,1189,2719,1446],{},[31,2716,1438],{},[60,2718,1442],{},[60,2720,1445],{},[28,2722,2723,1452],{},[31,2724,1451],{},[28,2726,2727,1458,2729,1462,2731,1466,2733,346,2735,1473,2737,1477],{},[31,2728,1457],{},[60,2730,1461],{},[60,2732,1465],{},[60,2734,1469],{},[60,2736,1472],{},[60,2738,1476],{},[28,2740,2741,1483,2743,1486,2745,209,2747,1491,2749],{},[31,2742,1482],{},[60,2744,200],{},[60,2746,208],{},[60,2748,212],{},[186,2750,2751],{"className":216,"code":1494,"language":218,"meta":194,"style":194},[60,2752,2753,2759,2767,2773,2781],{"__ignoreMap":194},[222,2754,2755,2757],{"class":224,"line":225},[222,2756,1501],{"class":228},[222,2758,233],{"class":232},[222,2760,2761,2763,2765],{"class":224,"line":236},[222,2762,1508],{"class":228},[222,2764,250],{"class":232},[222,2766,1513],{"class":253},[222,2768,2769,2771],{"class":224,"line":244},[222,2770,1518],{"class":228},[222,2772,233],{"class":232},[222,2774,2775,2777,2779],{"class":224,"line":261},[222,2776,1508],{"class":228},[222,2778,250],{"class":232},[222,2780,1529],{"class":276},[222,2782,2783,2785,2787],{"class":224,"line":269},[222,2784,1534],{"class":228},[222,2786,250],{"class":232},[222,2788,291],{"class":253},[28,2790,2791,1544,2793,1548,2795,1552,2797,1556,2799,1560,2801,1564],{},[31,2792,1543],{},[60,2794,1547],{},[60,2796,1551],{},[60,2798,1555],{},[60,2800,1559],{},[60,2802,1563],{},[75,2804,1568],{"id":1567},[186,2806,2808],{"className":2807,"code":1572,"language":191},[189],[60,2809,1572],{"__ignoreMap":194},[75,2811,1578],{"id":1577},[10,2813,1581],{},[186,2815,2816],{"className":437,"code":1584,"language":439,"meta":194,"style":194},[60,2817,2818,2826],{"__ignoreMap":194},[222,2819,2820,2822,2824],{"class":224,"line":225},[222,2821,1591],{"class":446},[222,2823,1594],{"class":456},[222,2825,1597],{"class":276},[222,2827,2828],{"class":224,"line":236},[222,2829,1602],{"class":257},[75,2831,1606],{"id":1605},[10,2833,1609],{},[158,2835,2836,2838,2840,2842,2844],{},[28,2837,1614],{},[28,2839,1617],{},[28,2841,1620],{},[28,2843,1623],{},[28,2845,1626],{},[10,2847,1629],{},[75,2849,1633],{"id":1632},[10,2851,1636,2852,1639],{},[60,2853,1227],{},[10,2855,1642],{},[20,2857,1646],{"id":1645},[10,2859,1649],{},[25,2861,2862,2864,2866,2868],{},[28,2863,1654],{},[28,2865,1657],{},[28,2867,1660],{},[28,2869,1663],{},[10,2871,1666],{},[1668,2873],{},[10,2875,2876],{},[1200,2877,1674,2878,310],{},[1211,2879,1678],{"href":1677},[1680,2881,1682],{},{"title":194,"searchDepth":236,"depth":236,"links":2883},[2884,2885,2892,2899,2900,2908],{"id":22,"depth":236,"text":23},{"id":72,"depth":236,"text":73,"children":2886},[2887,2888,2889,2890,2891],{"id":77,"depth":244,"text":78},{"id":152,"depth":244,"text":153},{"id":180,"depth":244,"text":181},{"id":318,"depth":244,"text":319},{"id":476,"depth":244,"text":477},{"id":574,"depth":236,"text":575,"children":2893},[2894,2895,2896,2897,2898],{"id":578,"depth":244,"text":579},{"id":685,"depth":244,"text":686},{"id":782,"depth":244,"text":783},{"id":893,"depth":244,"text":894},{"id":1148,"depth":244,"text":1149},{"id":1220,"depth":236,"text":1221},{"id":1307,"depth":236,"text":1308,"children":2901},[2902,2903,2904,2905,2906,2907],{"id":1314,"depth":244,"text":1315},{"id":1339,"depth":244,"text":1340},{"id":1567,"depth":244,"text":1568},{"id":1577,"depth":244,"text":1578},{"id":1605,"depth":244,"text":1606},{"id":1632,"depth":244,"text":1633},{"id":1645,"depth":236,"text":1646},{},{"title":5,"description":1711},1781554578448]