Skip to the content.

Tests Pypi

Try the jc web demo

JC is now available as an Ansible filter plugin in the community.general collection. See this blog post for an example.

JC

JSON CLI output utility

jc JSONifies the output of many CLI tools and file-types for easier parsing in scripts. See the Parsers section for supported commands and file-types.

dig example.com | jc --dig
[{"id":38052,"opcode":"QUERY","status":"NOERROR","flags":["qr","rd","ra"],"query_num":1,"answer_num":1,
"authority_num":0,"additional_num":1,"opt_pseudosection":{"edns":{"version":0,"flags":[],"udp":4096}},"question":
{"name":"example.com.","class":"IN","type":"A"},"answer":[{"name":"example.com.","class":"IN","type":"A","ttl":
39049,"data":"93.184.216.34"}],"query_time":49,"server":"2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)","when":
"Fri Apr 16 16:09:00 PDT 2021","rcvd":56,"when_epoch":1618614540,"when_epoch_utc":null}]

This allows further command-line processing of output with tools like jq or jello by piping commands:

$ dig example.com | jc --dig | jq -r '.[].answer[].data'
93.184.216.34

or using the alternative “magic” syntax:

$ jc dig example.com | jq -r '.[].answer[].data'
93.184.216.34

The jc parsers can also be used as python modules. In this case the output will be a python dictionary, or list of dictionaries, instead of JSON:

>>> import subprocess
>>> import jc.parsers.dig
>>> 
>>> cmd_output = subprocess.check_output(['dig', 'example.com'], text=True)
>>> data = jc.parsers.dig.parse(cmd_output)
>>>
>>> data
[{'id': 64612, 'opcode': 'QUERY', 'status': 'NOERROR', 'flags': ['qr', 'rd', 'ra'], 'query_num': 1, 'answer_num':
1, 'authority_num': 0, 'additional_num': 1, 'opt_pseudosection': {'edns': {'version': 0, 'flags': [], 'udp':
4096}}, 'question': {'name': 'example.com.', 'class': 'IN', 'type': 'A'}, 'answer': [{'name': 'example.com.',
'class': 'IN', 'type': 'A', 'ttl': 29658, 'data': '93.184.216.34'}], 'query_time': 52, 'server':
'2600:1700:bab0:d40::1#53(2600:1700:bab0:d40::1)', 'when': 'Fri Apr 16 16:13:00 PDT 2021', 'rcvd': 56,
'when_epoch': 1618614780, 'when_epoch_utc': None}]

Two representations of the data are available. The default representation uses a strict schema per parser and converts known numbers to int/float JSON values. Certain known values of None are converted to JSON null, known boolean values are converted, and, in some cases, additional semantic context fields are added.

To access the raw, pre-processed JSON, use the -r cli option or the raw=True function parameter in parse().

Schemas for each parser can be found at the documentation link beside each Parser below.

Release notes can be found here.

Why Would Anyone Do This!?

For more information on the motivations for this project, please see my blog post on Bringing the Unix Philosophy to the 21st Century.

See also:

Use Cases:

Installation

There are several ways to get jc. You can install via pip, OS package repositories, via DEB/RPM/MSI packaged binaries for linux and Windows, or by downloading the correct binary for your architecture and running it anywhere on your filesystem.

Pip (macOS, linux, unix, Windows)

pip3 install jc

OS Package Repositories

OS Command
Debian/Ubuntu linux apt-get install jc
Fedora linux dnf install jc
openSUSE linux zypper install jc
Arch linux pacman -S jc
NixOS linux nix-env -iA nixpkgs.jc or nix-env -iA nixos.jc
Guix System linux guix install jc
macOS brew install jc
FreeBSD portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean
Ansible filter plugin ansible-galaxy collection install community.general

For more packages and binaries, see the jc packaging site.

Usage

jc accepts piped input from STDIN and outputs a JSON representation of the previous command’s output to STDOUT.

COMMAND | jc PARSER [OPTIONS]

Alternatively, the “magic” syntax can be used by prepending jc to the command to be converted. Options can be passed to jc immediately before the command is given. (Note: command aliases and shell builtins are not supported)

jc [OPTIONS] COMMAND

The JSON output can be compact (default) or pretty formatted with the -p option.

Parsers

Options

Exit Codes

Any fatal errors within jc will generate an exit code of 100, otherwise the exit code will be 0. When using the “magic” syntax (e.g. jc ifconfig eth0), jc will store the exit code of the program being parsed and add it to the jc exit code. This way it is easier to determine if an error was from the parsed program or jc.

Consider the following examples using ifconfig:

ifconfig exit code jc exit code Combined exit code Interpretation
0 0 0 No errors
1 0 1 Error in ifconfig
0 100 100 Error in jc
1 100 101 Error in both ifconfig and jc

Setting Custom Colors via Environment Variable

You can specify custom colors via the JC_COLORS environment variable. The JC_COLORS environment variable takes four comma separated string values in the following format:

JC_COLORS=<keyname_color>,<keyword_color>,<number_color>,<string_color>

Where colors are: black, red, green, yellow, blue, magenta, cyan, gray, brightblack, brightred, brightgreen, brightyellow, brightblue, brightmagenta, brightcyan, white, or default

For example, to set to the default colors:

JC_COLORS=blue,brightblack,magenta,green

or

JC_COLORS=default,default,default,default

Streaming Parsers

Most parsers load all of the data from STDIN, parse it, then output the entire JSON document serially. There are some streaming parsers (e.g. ls-s and ping-s) that immediately start processing and outputing the data line-by-line as JSON Lines (aka NDJSON) while it is being received from STDIN. This can significantly reduce the amount of memory required to parse large amounts of command output (e.g. ls -lR /) and can sometimes process the data more quickly. Streaming parsers have slightly different behavior than standard parsers as outlined below.

Note: Streaming parsers cannot be used with the “magic” syntax

Ignoring Errors

You may want to ignore parsing errors when using streaming parsers since these may be used in long-lived processing pipelines and errors can break the pipe. To ignore parsing errors, use the -qq cli option or the ignore_exceptions=True argument with the parse() function. This will add a _jc_meta object to the JSON output with a success attribute. If success is true, then there were no issues parsing the line. If success is false, then a parsing issue was found and error and line fields will be added to include a short error description and the contents of the unparsable line, respectively:

Successfully parsed line with -qq option:

{
  "command_data": "data",
  "_jc_meta": {
    "success": true
  }
}

Unsuccessfully parsed line with -qq option:

{
  "_jc_meta": {
    "success": false,
    "error": "error message",
    "line": "original line data"
  }
}

Unbuffering Output

Most operating systems will buffer output that is being piped from process to process. The buffer is usually around 4KB. When viewing the output in the terminal the OS buffer is not engaged so output is immediately displayed on the screen. When piping multiple processes together, though, it may seem as if the output is hanging when the input data is very slow (e.g. ping):

$ ping 1.1.1.1 | jc --ping-s | jq
<slow output>

This is because the OS engages the 4KB buffer between jc and jq in this example. To display the data on the terminal in realtime, you can disable the buffer with the -u (unbuffer) cli option:

$ ping 1.1.1.1 | jc --ping-s -u | jq
{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"1","ttl":"128","time_ms":"24.6","duplicate":false}
{"type":"reply","pattern":null,"timestamp":null,"bytes":"64","response_ip":"1.1.1.1","icmp_seq":"2","ttl":"128","time_ms":"26.8","duplicate":false}
...

Note: Unbuffered output can be slower for large data streams.

Using Streaming Parsers as Python Modules

Streaming parsers accept any iterable object and return a generator iterator object allowing lazy processing of the data. The input data should iterate on lines of string data. Examples of good input data are sys.stdin or str.splitlines().

To use the generator object in your code, simply loop through it or use the next() builtin function:

import jc.parsers.ls_s

result = jc.parsers.ls_s.parse(ls_command_output.splitlines())
for item in result:
    print(item["filename"])

Custom Parsers

Custom local parser plugins may be placed in a jc/jcparsers folder in your local “App data directory”:

Local parser plugins are standard python module files. Use the jc/parsers/foo.py parser as a template and simply place a .py file in the jcparsers subfolder.

Local plugin filenames must be valid python module names, therefore must consist entirely of alphanumerics and start with a letter. Local plugins may override default parsers.

Note: The application data directory follows the XDG Base Directory Specification

Caveats

Locale

For best results set the LANG locale environment variable to C or en_US.UTF-8. For example, either by setting directly on the command-line:

$ LANG=C date | jc --date

or by exporting to the environment before running commands:

$ export LANG=C

Timezones

Some parsers have calculated epoch timestamp fields added to the output. Unless a timestamp field name has a _utc suffix it is considered naive. (i.e. based on the local timezone of the system the jc parser was run on).

If a UTC timezone can be detected in the text of the command output, the timestamp will be timezone aware and have a _utc suffix on the key name. (e.g. epoch_utc) No other timezones are supported for aware timestamps.

Compatibility

Some parsers like dig, xml, csv, etc. will work on any platform. Other parsers that convert platform-specific output will generate a warning message if they are run on an unsupported platform. To see all parser information, including compatibility, run jc -ap.

You may still use a parser on an unsupported platform - for example, you may want to parse a file with linux lsof output on an macOS or Windows laptop. In that case you can suppress the warning message with the -q cli option or the quiet=True function parameter in parse():

macOS:

cat lsof.out | jc --lsof -q

or Windows:

type lsof.out | jc --lsof -q

Tested on:

Contributions

Feel free to add/improve code or parsers! You can use the jc/parsers/foo.py or jc/parsers/foo_s.py (streaming) parsers as a template and submit your parser with a pull request.

Please see the Contributing Guidelines for more information.

Acknowledgments

Examples

Here are some examples of jc output. For more examples, see here or the parser documentation.

arp

arp | jc --arp -p          # or:  jc -p arp
[
  {
    "address": "gateway",
    "hwtype": "ether",
    "hwaddress": "00:50:56:f7:4a:fc",
    "flags_mask": "C",
    "iface": "ens33"
  },
  {
    "address": "192.168.71.1",
    "hwtype": "ether",
    "hwaddress": "00:50:56:c0:00:08",
    "flags_mask": "C",
    "iface": "ens33"
  },
  {
    "address": "192.168.71.254",
    "hwtype": "ether",
    "hwaddress": "00:50:56:fe:7a:b4",
    "flags_mask": "C",
    "iface": "ens33"
  }
]

CSV files

cat homes.csv
"Sell", "List", "Living", "Rooms", "Beds", "Baths", "Age", "Acres", "Taxes"
142, 160, 28, 10, 5, 3,  60, 0.28,  3167
175, 180, 18,  8, 4, 1,  12, 0.43,  4033
129, 132, 13,  6, 3, 1,  41, 0.33,  1471
...
cat homes.csv | jc --csv -p
[
  {
    "Sell": "142",
    "List": "160",
    "Living": "28",
    "Rooms": "10",
    "Beds": "5",
    "Baths": "3",
    "Age": "60",
    "Acres": "0.28",
    "Taxes": "3167"
  },
  {
    "Sell": "175",
    "List": "180",
    "Living": "18",
    "Rooms": "8",
    "Beds": "4",
    "Baths": "1",
    "Age": "12",
    "Acres": "0.43",
    "Taxes": "4033"
  },
  {
    "Sell": "129",
    "List": "132",
    "Living": "13",
    "Rooms": "6",
    "Beds": "3",
    "Baths": "1",
    "Age": "41",
    "Acres": "0.33",
    "Taxes": "1471"
  }
]

/etc/hosts file

cat /etc/hosts | jc --hosts -p
[
  {
    "ip": "127.0.0.1",
    "hostname": [
      "localhost"
    ]
  },
  {
    "ip": "::1",
    "hostname": [
      "ip6-localhost",
      "ip6-loopback"
    ]
  },
  {
    "ip": "fe00::0",
    "hostname": [
      "ip6-localnet"
    ]
  }
]

ifconfig

ifconfig | jc --ifconfig -p          # or:  jc -p ifconfig
[
  {
    "name": "ens33",
    "flags": 4163,
    "state": [
      "UP",
      "BROADCAST",
      "RUNNING",
      "MULTICAST"
    ],
    "mtu": 1500,
    "ipv4_addr": "192.168.71.137",
    "ipv4_mask": "255.255.255.0",
    "ipv4_bcast": "192.168.71.255",
    "ipv6_addr": "fe80::c1cb:715d:bc3e:b8a0",
    "ipv6_mask": 64,
    "ipv6_scope": "0x20",
    "mac_addr": "00:0c:29:3b:58:0e",
    "type": "Ethernet",
    "rx_packets": 8061,
    "rx_bytes": 1514413,
    "rx_errors": 0,
    "rx_dropped": 0,
    "rx_overruns": 0,
    "rx_frame": 0,
    "tx_packets": 4502,
    "tx_bytes": 866622,
    "tx_errors": 0,
    "tx_dropped": 0,
    "tx_overruns": 0,
    "tx_carrier": 0,
    "tx_collisions": 0,
    "metric": null
  }
]

INI files

cat example.ini
[DEFAULT]
ServerAliveInterval = 45
Compression = yes
CompressionLevel = 9
ForwardX11 = yes

[bitbucket.org]
User = hg

[topsecret.server.com]
Port = 50022
ForwardX11 = no
cat example.ini | jc --ini -p
{
  "bitbucket.org": {
    "serveraliveinterval": "45",
    "compression": "yes",
    "compressionlevel": "9",
    "forwardx11": "yes",
    "user": "hg"
  },
  "topsecret.server.com": {
    "serveraliveinterval": "45",
    "compression": "yes",
    "compressionlevel": "9",
    "forwardx11": "no",
    "port": "50022"
  }
}

ls

$ ls -l /usr/bin | jc --ls -p          # or:  jc -p ls -l /usr/bin
[
  {
    "filename": "apropos",
    "link_to": "whatis",
    "flags": "lrwxrwxrwx.",
    "links": 1,
    "owner": "root",
    "group": "root",
    "size": 6,
    "date": "Aug 15 10:53"
  },
  {
    "filename": "ar",
    "flags": "-rwxr-xr-x.",
    "links": 1,
    "owner": "root",
    "group": "root",
    "size": 62744,
    "date": "Aug 8 16:14"
  },
  {
    "filename": "arch",
    "flags": "-rwxr-xr-x.",
    "links": 1,
    "owner": "root",
    "group": "root",
    "size": 33080,
    "date": "Aug 19 23:25"
  }
]

netstat

netstat -apee | jc --netstat -p          # or:  jc -p netstat -apee
[
  {
    "proto": "tcp",
    "recv_q": 0,
    "send_q": 0,
    "local_address": "localhost",
    "foreign_address": "0.0.0.0",
    "state": "LISTEN",
    "user": "systemd-resolve",
    "inode": 26958,
    "program_name": "systemd-resolve",
    "kind": "network",
    "pid": 887,
    "local_port": "domain",
    "foreign_port": "*",
    "transport_protocol": "tcp",
    "network_protocol": "ipv4"
  },
  {
    "proto": "tcp6",
    "recv_q": 0,
    "send_q": 0,
    "local_address": "[::]",
    "foreign_address": "[::]",
    "state": "LISTEN",
    "user": "root",
    "inode": 30510,
    "program_name": "sshd",
    "kind": "network",
    "pid": 1186,
    "local_port": "ssh",
    "foreign_port": "*",
    "transport_protocol": "tcp",
    "network_protocol": "ipv6"
  },
  {
    "proto": "udp",
    "recv_q": 0,
    "send_q": 0,
    "local_address": "localhost",
    "foreign_address": "0.0.0.0",
    "state": null,
    "user": "systemd-resolve",
    "inode": 26957,
    "program_name": "systemd-resolve",
    "kind": "network",
    "pid": 887,
    "local_port": "domain",
    "foreign_port": "*",
    "transport_protocol": "udp",
    "network_protocol": "ipv4"
  },
  {
    "proto": "raw6",
    "recv_q": 0,
    "send_q": 0,
    "local_address": "[::]",
    "foreign_address": "[::]",
    "state": "7",
    "user": "systemd-network",
    "inode": 27001,
    "program_name": "systemd-network",
    "kind": "network",
    "pid": 867,
    "local_port": "ipv6-icmp",
    "foreign_port": "*",
    "transport_protocol": null,
    "network_protocol": "ipv6"
  },
  {
    "proto": "unix",
    "refcnt": 2,
    "flags": null,
    "type": "DGRAM",
    "state": null,
    "inode": 33322,
    "program_name": "systemd",
    "path": "/run/user/1000/systemd/notify",
    "kind": "socket",
    "pid": 1607
  }
]

/etc/passwd file

cat /etc/passwd | jc --passwd -p
[
  {
    "username": "root",
    "password": "*",
    "uid": 0,
    "gid": 0,
    "comment": "System Administrator",
    "home": "/var/root",
    "shell": "/bin/sh"
  },
  {
    "username": "daemon",
    "password": "*",
    "uid": 1,
    "gid": 1,
    "comment": "System Services",
    "home": "/var/root",
    "shell": "/usr/bin/false"
  }
]

ping

ping 8.8.8.8 -c 3 | jc --ping -p          # or:  jc -p ping 8.8.8.8 -c 3
{
  "destination_ip": "8.8.8.8",
  "data_bytes": 56,
  "pattern": null,
  "destination": "8.8.8.8",
  "packets_transmitted": 3,
  "packets_received": 3,
  "packet_loss_percent": 0.0,
  "duplicates": 0,
  "time_ms": 2005.0,
  "round_trip_ms_min": 23.835,
  "round_trip_ms_avg": 30.46,
  "round_trip_ms_max": 34.838,
  "round_trip_ms_stddev": 4.766,
  "responses": [
    {
      "type": "reply",
      "timestamp": null,
      "bytes": 64,
      "response_ip": "8.8.8.8",
      "icmp_seq": 1,
      "ttl": 118,
      "time_ms": 23.8,
      "duplicate": false
    },
    {
      "type": "reply",
      "timestamp": null,
      "bytes": 64,
      "response_ip": "8.8.8.8",
      "icmp_seq": 2,
      "ttl": 118,
      "time_ms": 34.8,
      "duplicate": false
    },
    {
      "type": "reply",
      "timestamp": null,
      "bytes": 64,
      "response_ip": "8.8.8.8",
      "icmp_seq": 3,
      "ttl": 118,
      "time_ms": 32.7,
      "duplicate": false
    }
  ]
}

ps

ps axu | jc --ps -p          # or:  jc -p ps axu
[
  {
    "user": "root",
    "pid": 1,
    "cpu_percent": 0.0,
    "mem_percent": 0.1,
    "vsz": 128072,
    "rss": 6784,
    "tty": null,
    "stat": "Ss",
    "start": "Nov09",
    "time": "0:08",
    "command": "/usr/lib/systemd/systemd --switched-root --system --deserialize 22"
  },
  {
    "user": "root",
    "pid": 2,
    "cpu_percent": 0.0,
    "mem_percent": 0.0,
    "vsz": 0,
    "rss": 0,
    "tty": null,
    "stat": "S",
    "start": "Nov09",
    "time": "0:00",
    "command": "[kthreadd]"
  },
  {
    "user": "root",
    "pid": 4,
    "cpu_percent": 0.0,
    "mem_percent": 0.0,
    "vsz": 0,
    "rss": 0,
    "tty": null,
    "stat": "S<",
    "start": "Nov09",
    "time": "0:00",
    "command": "[kworker/0:0H]"
  }
]

traceroute

traceroute -m 2 8.8.8.8 | jc --traceroute -p          # or:  jc -p traceroute -m 2 8.8.8.8
{
  "destination_ip": "8.8.8.8",
  "destination_name": "8.8.8.8",
  "hops": [
    {
      "hop": 1,
      "probes": [
        {
          "annotation": null,
          "asn": null,
          "ip": "192.168.1.254",
          "name": "dsldevice.local.net",
          "rtt": 6.616
        },
        {
          "annotation": null,
          "asn": null,
          "ip": "192.168.1.254",
          "name": "dsldevice.local.net",
          "rtt": 6.413
        },
        {
          "annotation": null,
          "asn": null,
          "ip": "192.168.1.254",
          "name": "dsldevice.local.net",
          "rtt": 6.308
        }
      ]
    },
    {
      "hop": 2,
      "probes": [
        {
          "annotation": null,
          "asn": null,
          "ip": "76.220.24.1",
          "name": "76-220-24-1.lightspeed.sntcca.sbcglobal.net",
          "rtt": 29.367
        },
        {
          "annotation": null,
          "asn": null,
          "ip": "76.220.24.1",
          "name": "76-220-24-1.lightspeed.sntcca.sbcglobal.net",
          "rtt": 40.197
        },
        {
          "annotation": null,
          "asn": null,
          "ip": "76.220.24.1",
          "name": "76-220-24-1.lightspeed.sntcca.sbcglobal.net",
          "rtt": 29.162
        }
      ]
    }
  ]
}

uptime

uptime | jc --uptime -p          # or:  jc -p uptime
{
  "time": "11:35",
  "uptime": "3 days, 4:03",
  "users": 5,
  "load_1m": 1.88,
  "load_5m": 2.0,
  "load_15m": 1.94,
  "time_hour": 11,
  "time_minute": 35,
  "time_second": null,
  "uptime_days": 3,
  "uptime_hours": 4,
  "uptime_minutes": 3,
  "uptime_total_seconds": 273780
}

XML files

cat cd_catalog.xml
<?xml version="1.0" encoding="UTF-8"?>
<CATALOG>
  <CD>
    <TITLE>Empire Burlesque</TITLE>
    <ARTIST>Bob Dylan</ARTIST>
    <COUNTRY>USA</COUNTRY>
    <COMPANY>Columbia</COMPANY>
    <PRICE>10.90</PRICE>
    <YEAR>1985</YEAR>
  </CD>
  <CD>
    <TITLE>Hide your heart</TITLE>
    <ARTIST>Bonnie Tyler</ARTIST>
    <COUNTRY>UK</COUNTRY>
    <COMPANY>CBS Records</COMPANY>
    <PRICE>9.90</PRICE>
    <YEAR>1988</YEAR>
  </CD>
  ...
cat cd_catalog.xml | jc --xml -p
{
  "CATALOG": {
    "CD": [
      {
        "TITLE": "Empire Burlesque",
        "ARTIST": "Bob Dylan",
        "COUNTRY": "USA",
        "COMPANY": "Columbia",
        "PRICE": "10.90",
        "YEAR": "1985"
      },
      {
        "TITLE": "Hide your heart",
        "ARTIST": "Bonnie Tyler",
        "COUNTRY": "UK",
        "COMPANY": "CBS Records",
        "PRICE": "9.90",
        "YEAR": "1988"
      }
    ]
  }
}

YAML files

cat istio.yaml 
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  namespace: "default"
spec:
  peers:
  - mtls: {}
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "default"
spec:
  host: "*.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
cat istio.yaml | jc --yaml -p
[
  {
    "apiVersion": "authentication.istio.io/v1alpha1",
    "kind": "Policy",
    "metadata": {
      "name": "default",
      "namespace": "default"
    },
    "spec": {
      "peers": [
        {
          "mtls": {}
        }
      ]
    }
  },
  {
    "apiVersion": "networking.istio.io/v1alpha3",
    "kind": "DestinationRule",
    "metadata": {
      "name": "default",
      "namespace": "default"
    },
    "spec": {
      "host": "*.default.svc.cluster.local",
      "trafficPolicy": {
        "tls": {
          "mode": "ISTIO_MUTUAL"
        }
      }
    }
  }
]

© 2019-2021 Kelly Brazil