Review a failed systemd service: Capture status, Read journal, Identify cause, Approve retry

What systemctl reset-failed Actually Clears Before You Retry a Service

Preserve systemd failure evidence, understand start-limit counters, and distinguish clearing a failed state from repairing or restarting a Linux service.

systemctl reset-failed clears a recorded failure state and resets associated counters; it does not repair a service or start it. Before using it, preserve the failed unit’s status and journal evidence, identify why the application stopped, and decide whether a retry is safe. A cleared failure marker is not evidence that the application is healthy.

Capture the failure before clearing its marker

These examples use example.service as a placeholder. Replace it with the exact unit name on your own system. Start with read-only inspection:

systemctl status example.service --no-pager --full
systemctl show example.service --property=LoadState,ActiveState,SubState,Result,ExecMainCode,ExecMainStatus,NRestarts
journalctl --unit=example.service --since='1 hour ago' --no-pager

The first command provides a human-readable summary. The second exposes selected properties suitable for recording and comparison. The third reads recent journal messages; adjust the time window to include the actual failure. Some journal entries require additional permissions, so use your organisation’s approved access rather than changing log permissions.

Record the timestamp, host, unit name and relevant output in the incident ticket. Redact tokens, customer data and credentials before sharing logs. Status output can include only a small recent portion of the journal, and it describes the current or most recent invocation still retained by systemd. Use the journal to investigate earlier runs rather than assuming the visible summary contains the full incident.

Separate application failure from retry throttling

A service might fail because its configuration is invalid, its executable is unavailable, a dependency is not ready, or its process exits unsuccessfully. A rapid sequence of failed starts can also hit the configured start-rate limit. These are related observations, not interchangeable diagnoses.

For example, imagine a fictional service repeatedly exits after reporting an invalid configuration key. Later messages say that start requests are being repeated too quickly. Resetting the limiter may permit another attempt, but that attempt will encounter the same invalid key until the configuration problem is fixed. Conversely, do not assume that every failed service has hit a rate limit: read the actual messages.

If you recently edited a unit definition and suspect systemd still has different loaded settings, use the related disk-versus-loaded service settings workflow. Application configuration files and systemd unit definitions are different layers; a manager reload is not a universal application repair.

Know the scope of reset-failed

The official systemctl manual explains that resetting a failed unit also clears its start-rate-limit counter and, for a service, its restart counter. Those counters can be useful evidence, which is another reason to capture them first.

For an approved change on one known unit, the targeted command is:

sudo systemctl reset-failed example.service

Do not omit the unit name casually. Without a name, the command resets failed states across units rather than just the one under investigation. A system-wide cleanup can make an incident dashboard look quieter while obscuring which failures were still awaiting review. There is no useful rollback that reconstructs the original counters merely by running another reset command; preserve the pre-change evidence.

Use the correct manager scope too. These examples address system services. A user service belongs to a user manager and normally needs corresponding systemctl --user and journalctl --user-unit= commands in that user’s context, not an arbitrary root shell operating against the wrong manager.

Retry only after the cause and side effects are understood

Starting a service is a separate state-changing operation. Do not automatically append a start command to reset-failed in an incident script. A unit may send queued mail, repeat a data import, or perform other work as soon as it starts. Confirm duplicate-work risks, dependencies and change approval first.

After an authorised repair and retry, inspect the same status properties and new journal entries again. Check the application’s own health signal as well: an active process is not proof that a website responds correctly or that a job produced its intended output. A short-lived oneshot service can legitimately become inactive after successful work, so interpret its service type and result rather than demanding a permanently running process.

Quick questions before closing the incident

  • Was the original failure and its time window preserved?
  • Was a root cause identified separately from any rate-limit symptom?
  • Was only the intended unit reset under the correct manager?
  • Was the retry authorised and checked for repeated side effects?
  • Did both systemd evidence and application-level checks support recovery?

Does clearing failed state erase every journal message? No; resetting the unit state and querying historical logs are different operations. Journal availability still depends on retention and access.

Does reset-failed start the service? No. Treat any subsequent start as a separate operational decision, not an automatic consequence of clearing a marker.

Educational use only. Test carefully before production use, and preserve configuration backups and incident evidence before changes. The site owner is not responsible for misuse, damage, data loss or production issues.

Leave a Reply

Your email address will not be published. Required fields are marked *