Linux CLI + Go: Automation Commands Every Developer Should Master

Think of your Linux system as a factory floor. The shell is the conveyor belt — it moves data from one station to the next. Each command is a machine that transforms the material passing through it. A skilled factory engineer does not rebuild a machine for every task. They wire existing machines together, and they build new specialized machines only when the standard ones fall short.

This is the mindset behind automating with Linux and Go. Most day-to-day automation is already handled by find, xargs, awk, cron, and systemd. When those tools hit their limits — when you need typed config, structured output, retries, or HTTP calls — you write a Go binary and plug it into the same conveyor belt. The result is a system that is fast, auditable, and runs without a runtime.

This post covers both layers. First, the Linux commands worth internalizing. Then, the Go patterns for building CLI tools that plug into them.

---

The Foundation: Pipes and Redirects

Before any specific command, the mechanism that makes all of them composable deserves a clear explanation. Every Linux process has three standard streams: stdin (input), stdout (output), and stderr (errors). The pipe operator | connects stdout of one process to stdin of the next. Redirects (>, >>, 2>) send streams to files.

# stdout to file, stderr discarded
go build ./... > build.log 2>/dev/null

# stderr to same file as stdout
go test ./... > test.log 2>&1

# append to existing file
echo "$(date): job started" >> /var/log/my-app/cron.log

# stdin from file
psql -U app -d mydb < schema.sql

Understanding these primitives means you can compose any chain of commands into a pipeline. Every tool in this post operates through these three streams.

---

find: Locate Files with Precision

find traverses a directory tree and applies predicates. It is more capable than most developers use it for.

The basic form is find <path> <predicates>. Predicates combine with implicit AND by default.

# find all .go files modified in the last 24 hours
find ./src -name "*.go" -mtime -1

# find files larger than 10MB
find /var/log -name "*.log" -size +10M

# find and delete temp files older than 7 days
find /tmp -name "*.tmp" -mtime +7 -delete

# find directories only
find . -type d -name "vendor"

# find files not owned by current user
find /var/app -not -user $(whoami) -type f

The -exec flag runs a command on each result. The {} placeholder is the matched file, and \; ends the command.

# compress each old log file
find /var/log -name "*.log" -mtime +30 -exec gzip {} \;

# print file size and name for all binaries
find ./bin -type f -exec du -sh {} \;

Use + instead of \; to pass all matches as arguments at once — more efficient when the command accepts multiple inputs:

find . -name "*.go" -exec gofmt -w {} +

---

xargs: Convert Lines to Arguments

xargs reads lines from stdin and passes them as arguments to a command. It bridges the gap between commands that output file lists and commands that accept file arguments.

# lint all Go files found by find
find . -name "*.go" | xargs golangci-lint run

# delete all files listed in a text file
cat files-to-delete.txt | xargs rm -f

# run with 4 parallel processes (-P)
find ./images -name "*.png" | xargs -P 4 -I{} convert {} -resize 800x {}-resized.png

The -I{} flag defines a placeholder, allowing you to control where the argument appears in the command. The -P flag controls parallelism. Together, they turn a sequential list into a parallel job queue — without writing a scheduler.

# run go test in 4 packages in parallel
echo -e "pkg/auth\npkg/user\npkg/billing\npkg/notify" | xargs -P 4 -I{} go test ./{} -v

---

awk: Transform Structured Text

awk processes text line by line, splitting each line into fields. It is the right tool when you need to extract columns from command output, compute sums, or reformat reports.

# print second field (process name) from ps output
ps aux | awk '{print $11}'

# sum the size column from du output
du -sh ./logs/*.log | awk '{sum += $1} END {print "Total:", sum}'

# print lines where memory usage exceeds 100MB
ps aux | awk '$6 > 102400 {print $2, $11, $6/1024 "MB"}'

# extract only the HTTP status code from nginx log
cat access.log | awk '{print $9}' | sort | uniq -c | sort -rn

awk also reads files directly. This is useful when processing structured output stored to disk:

# extract failed builds from a CI log file
awk '/FAIL/{print NR, $0}' ci-output.log

---

sed: In-place Text Substitution

sed applies substitution patterns to streams of text. The most common use is search-and-replace, but it also handles line insertion, deletion, and extraction.

# replace all occurrences of "localhost" with the real hostname
sed 's/localhost/db.internal/g' config.template > config.env

# in-place edit (modify file directly)
sed -i 's/DEBUG=true/DEBUG=false/g' .env.production

# delete lines matching a pattern
sed -i '/^#/d' config.env   # remove comment lines

# print only lines 10 to 20
sed -n '10,20p' large-file.log

# insert a line before a match
sed -i '/^ENV=/i # Environment configuration' .env

Combining sed with find and xargs gives you a refactoring tool:

# rename a package across all Go files
find . -name "*.go" | xargs sed -i 's/package oldname/package newname/g'

---

cron: Scheduled Jobs

cron runs commands on a schedule defined in a crontab. The schedule format has five fields: minute, hour, day-of-month, month, day-of-week.

# edit the current user's crontab
crontab -e

# list current crontab
crontab -l

Common schedule patterns:

# run at 2:30 AM every day
30 2 * * * /usr/local/bin/backup.sh

# run every 15 minutes
*/15 * * * * /usr/local/bin/health-check

# run every Monday at 9 AM
0 9 * * 1 /usr/local/bin/weekly-report

# run on the first day of each month
0 0 1 * * /usr/local/bin/monthly-cleanup

A cron job that calls a Go binary looks exactly the same. Compile the binary and point to it:

# /etc/cron.d/my-app-jobs
0 */6 * * * app-user /usr/local/bin/my-app sync --config /etc/my-app/config.yaml >> /var/log/my-app/sync.log 2>&1

Key cron hygiene practices:

• Always use absolute paths. cron does not inherit your shell’s PATH.
• Redirect both stdout and stderr to a log file with >> log 2>&1.
• Test commands manually before adding to crontab.

---

systemd Timers: cron with Supervision

systemd timers are the modern alternative to cron. They integrate with the service manager, provide logging via journalctl, and support dependency ordering.

A timer requires two unit files: a .service and a .timer.

# /etc/systemd/system/sync-data.service
[Unit]
Description=Sync data job

[Service]
Type=oneshot
User=app
ExecStart=/usr/local/bin/my-app sync --config /etc/my-app/config.yaml
StandardOutput=journal
StandardError=journal

# /etc/systemd/system/sync-data.timer
[Unit]
Description=Run sync-data every 6 hours

[Timer]
OnCalendar=*-*-* 00,06,12,18:00:00
Persistent=true

[Install]
WantedBy=timers.target

# enable and start
systemctl enable --now sync-data.timer

# check status
systemctl status sync-data.timer

# view logs
journalctl -u sync-data.service -f

The Persistent=true flag means if the system was off during the scheduled time, the timer runs immediately on next boot. This makes it safer than cron for jobs that must not be skipped.

---

inotifywait: React to File System Events

inotifywait (from the inotify-tools package) watches for file system events — creation, modification, deletion — and outputs them to stdout. This is the foundation for file watchers, hot-reload scripts, and reactive pipelines.

# watch a directory for any changes
inotifywait -m -r /var/app/uploads

# watch for new files only, output in a parseable format
inotifywait -m -e create --format '%w%f' /var/app/uploads

# trigger a command when a config file changes
inotifywait -m -e modify /etc/my-app/config.yaml | while read path event file; do
    echo "Config changed, reloading..."
    systemctl reload my-app
done

Wrapping this in a shell loop gives you a reactive trigger. The pattern is: watch for event, pipe to while read, react.

# process each uploaded file as it arrives
inotifywait -m -e create --format '%f' /var/app/uploads | while read filename; do
    /usr/local/bin/process-upload "$filename"
done

---

Building Go CLI Tools

The Linux commands above handle data transformation. Go handles logic that is too complex for shell — HTTP calls, structured config, retries, typed errors, concurrent workers. The key is building Go binaries that behave like proper Unix tools: they read from stdin, write to stdout, accept flags, and exit with meaningful codes.

Reading Flags and Arguments

Go’s standard library flag package covers simple tools. For complex CLIs with subcommands, use github.com/spf13/cobra.

// cmd/sync/main.go
package main

import (
	"flag"
	"fmt"
	"log"
	"os"
)

func main() {
	configPath := flag.String("config", "/etc/my-app/config.yaml", "path to config file")
	dryRun := flag.Bool("dry-run", false, "print actions without executing")
	verbose := flag.Bool("v", false, "verbose output")
	flag.Parse()

	if *verbose {
		log.SetFlags(log.Ldate | log.Ltime | log.Lshortfile)
	} else {
		log.SetFlags(0)
	}

	cfg, err := loadConfig(*configPath)
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}

	if err := run(cfg, *dryRun); err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
}

Exit code 1 signals failure to the calling shell. Cron, systemd, and xargs all check exit codes and act accordingly.

Reading from Stdin and Writing to Stdout

A Go binary that reads from stdin and writes to stdout composes perfectly with pipes.

// cmd/filter-logs/main.go — reads log lines from stdin, writes matching lines to stdout
package main

import (
	"bufio"
	"flag"
	"fmt"
	"os"
	"strings"
)

func main() {
	level := flag.String("level", "ERROR", "log level to filter for")
	flag.Parse()

	scanner := bufio.NewScanner(os.Stdin)
	for scanner.Scan() {
		line := scanner.Text()
		if strings.Contains(line, "["+*level+"]") {
			fmt.Println(line)
		}
	}

	if err := scanner.Err(); err != nil {
		fmt.Fprintf(os.Stderr, "read error: %v\n", err)
		os.Exit(1)
	}
}

Build and use it in a pipeline:

go build -o /usr/local/bin/filter-logs ./cmd/filter-logs
cat /var/log/my-app/app.log | filter-logs --level=ERROR | mail -s "Errors" ops@company.com

Structured Config with YAML

Hard-coded values in CLI tools create problems when the same binary runs in dev, staging, and production. Read config from a file and override with environment variables.

// internal/config/config.go
package config

import (
	"fmt"
	"os"

	"gopkg.in/yaml.v3"
)

type Config struct {
	Database DatabaseConfig `yaml:"database"`
	Storage  StorageConfig  `yaml:"storage"`
	Log      LogConfig      `yaml:"log"`
}

type DatabaseConfig struct {
	DSN         string `yaml:"dsn"`
	MaxConns    int    `yaml:"max_conns"`
}

type StorageConfig struct {
	BasePath string `yaml:"base_path"`
	MaxSizeMB int   `yaml:"max_size_mb"`
}

type LogConfig struct {
	Level  string `yaml:"level"`
	Format string `yaml:"format"`
}

func Load(path string) (*Config, error) {
	data, err := os.ReadFile(path)
	if err != nil {
		return nil, fmt.Errorf("reading config: %w", err)
	}

	// expand environment variables in the YAML
	expanded := os.ExpandEnv(string(data))

	var cfg Config
	if err := yaml.Unmarshal([]byte(expanded), &cfg); err != nil {
		return nil, fmt.Errorf("parsing config: %w", err)
	}

	return &cfg, nil
}

The os.ExpandEnv call replaces ${VAR} placeholders in the YAML with actual environment variables. A single config file works across environments by changing environment variables, not the file.

# /etc/my-app/config.yaml
database:
  dsn: "${DATABASE_URL}"
  max_conns: 10
storage:
  base_path: "/var/app/data"
  max_size_mb: 1024
log:
  level: "${LOG_LEVEL:-info}"
  format: "json"

Concurrent Workers with a WaitGroup

Go’s goroutines make parallel processing trivial. The pattern for processing a list of items concurrently uses a worker pool: a buffered channel for jobs, a fixed number of goroutines consuming from it, and a sync.WaitGroup to wait for completion.

// internal/worker/pool.go
package worker

import (
	"sync"
)

type Job func() error

func RunPool(jobs []Job, concurrency int) []error {
	jobCh := make(chan Job, len(jobs))
	for _, j := range jobs {
		jobCh <- j
	}
	close(jobCh)

	var mu sync.Mutex
	var errs []error
	var wg sync.WaitGroup

	for range concurrency {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for job := range jobCh {
				if err := job(); err != nil {
					mu.Lock()
					errs = append(errs, err)
					mu.Unlock()
				}
			}
		}()
	}

	wg.Wait()
	return errs
}

Usage — processing a list of files with 8 concurrent workers:

files, _ := filepath.Glob("/var/app/uploads/*.csv")
jobs := make([]worker.Job, len(files))
for i, f := range files {
    path := f
    jobs[i] = func() error {
        return processCSV(path)
    }
}

errs := worker.RunPool(jobs, 8)
if len(errs) > 0 {
    for _, e := range errs {
        log.Printf("error: %v", e)
    }
    os.Exit(1)
}

File Watcher in Go

Instead of depending on inotifywait, you can write a file watcher directly in Go using github.com/fsnotify/fsnotify. This works cross-platform and integrates cleanly with your existing config and error handling.

// cmd/watch/main.go
package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/fsnotify/fsnotify"
)

func main() {
	dir := "/var/app/uploads"
	if len(os.Args) > 1 {
		dir = os.Args[1]
	}

	watcher, err := fsnotify.NewWatcher()
	if err != nil {
		log.Fatalf("creating watcher: %v", err)
	}
	defer watcher.Close()

	if err := watcher.Add(dir); err != nil {
		log.Fatalf("watching %s: %v", dir, err)
	}

	fmt.Printf("watching %s\n", dir)

	for {
		select {
		case event, ok := <-watcher.Events:
			if !ok {
				return
			}
			if event.Has(fsnotify.Create) {
				handleNewFile(event.Name)
			}
		case err, ok := <-watcher.Errors:
			if !ok {
				return
			}
			log.Printf("watcher error: %v", err)
		}
	}
}

func handleNewFile(path string) {
	ext := filepath.Ext(path)
	fmt.Printf("new file: %s (ext: %s)\n", path, ext)
	// process based on extension
}

Run this as a systemd service (type simple instead of oneshot) and it runs continuously, reacting to every new file without polling.

Graceful Shutdown

A CLI tool that runs as a long-lived service must handle SIGTERM and SIGINT cleanly — draining in-progress work before exiting.

// cmd/service/main.go
package main

import (
	"context"
	"log"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	svc := NewService()
	go svc.Run(ctx)

	quit := make(chan os.Signal, 1)
	signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
	<-quit

	log.Println("shutting down...")
	cancel()

	// give workers up to 10 seconds to finish
	shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer shutdownCancel()

	if err := svc.Shutdown(shutdownCtx); err != nil {
		log.Printf("shutdown error: %v", err)
		os.Exit(1)
	}

	log.Println("stopped cleanly")
}

When systemctl stop my-service runs, systemd sends SIGTERM. This pattern ensures in-flight work completes before the process exits.

---

Putting It Together: A Real Automation Pipeline

The individual tools are more useful when you see how they combine into a real workflow. Consider a data pipeline that runs every 6 hours: it downloads reports, processes each CSV, uploads results, and sends a summary.

The systemd timer calls the Go binary. The Go binary reads config from YAML, processes files with a worker pool, writes structured logs that journalctl captures, and exits with a non-zero code on failure — which systemd marks as a failed unit, triggering an alert.

# Makefile — build and install the binary
.PHONY: build install

build:
	go build -ldflags="-s -w" -o ./bin/pipeline ./cmd/pipeline

install: build
	install -m 755 ./bin/pipeline /usr/local/bin/pipeline
	install -m 644 ./deploy/pipeline.service /etc/systemd/system/
	install -m 644 ./deploy/pipeline.timer /etc/systemd/system/
	systemctl daemon-reload
	systemctl enable --now pipeline.timer

// cmd/pipeline/main.go
package main

import (
	"context"
	"flag"
	"fmt"
	"log/slog"
	"os"
	"time"

	"myorg/pipeline/internal/config"
	"myorg/pipeline/internal/downloader"
	"myorg/pipeline/internal/processor"
	"myorg/pipeline/internal/uploader"
)

func main() {
	configPath := flag.String("config", "/etc/pipeline/config.yaml", "config file")
	flag.Parse()

	logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

	cfg, err := config.Load(*configPath)
	if err != nil {
		logger.Error("config load failed", "error", err)
		os.Exit(1)
	}

	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
	defer cancel()

	start := time.Now()
	logger.Info("pipeline started")

	files, err := downloader.FetchReports(ctx, cfg.Source)
	if err != nil {
		logger.Error("download failed", "error", err)
		os.Exit(1)
	}

	logger.Info("downloaded reports", "count", len(files))

	results, err := processor.RunAll(ctx, files, cfg.Processing.Concurrency)
	if err != nil {
		logger.Error("processing failed", "error", err)
		os.Exit(1)
	}

	if err := uploader.Push(ctx, results, cfg.Destination); err != nil {
		logger.Error("upload failed", "error", err)
		os.Exit(1)
	}

	logger.Info("pipeline complete",
		"duration", time.Since(start).Round(time.Second),
		"processed", len(results),
	)
}

The slog.NewJSONHandler output goes to stdout, which systemd captures in the journal. Query it with:

journalctl -u pipeline.service --since "6 hours ago" -o json | jq '.MESSAGE | fromjson | select(.level == "ERROR")'

---

The Makefile as Automation Hub

A Makefile at the project root is the entry point for all local automation. It documents available commands and ensures they run the same way in every environment.

.PHONY: build test lint fmt install clean run

BIN := ./bin/my-app
SRC := ./cmd/my-app

build:
	go build -ldflags="-s -w" -o $(BIN) $(SRC)

test:
	go test ./... -race -count=1

lint:
	golangci-lint run ./...

fmt:
	gofmt -w .
	goimports -w .

install: build
	install -m 755 $(BIN) /usr/local/bin/my-app

clean:
	rm -f $(BIN)
	find . -name "*.log" -delete

run: build
	$(BIN) --config ./config/local.yaml

# generate templ files before building
generate:
	templ generate
	go generate ./...

Calling make build always produces the same result. No one has to remember the ldflags. New team members run make install and they are done.

---

Measuring What You Automate

Automation that runs silently is automation you cannot trust. Every scheduled job should produce observable output: log lines with timestamps and context, structured fields that tools like jq can filter, and exit codes that supervision systems can monitor.

The Go log/slog package (standard since Go 1.21) produces structured JSON logs with no third-party dependency:

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelInfo,
}))

logger.Info("job started", "job", "sync", "env", os.Getenv("APP_ENV"))
logger.Error("database unreachable", "error", err, "dsn", cfg.Database.DSN)

Output you can query:

journalctl -u sync-data.service -o json \
  | jq 'select(.PRIORITY == "3") | .MESSAGE | fromjson'

The discipline of observable automation is what separates scripts that work once from systems that you trust at 3 AM when an alert fires.

---

The Linux command line and Go occupy different layers of the same stack. The shell excels at wiring existing tools together — it is fast to write and easy to read. Go excels at complex logic — it is type-checked, testable, and compiles to a single binary. The developer who internalizes both layers does not choose between them. They reach for the shell when shell is enough, reach for Go when the logic demands it, and build pipelines where both collaborate.

The most maintainable automation is the kind where you can trace exactly what happened, when it happened, and why it succeeded or failed — without running it again.