Input and Output

Contents

• 1. Introduction
• 2. Standard Input
• 3. Streams
• 4. Reading and Writing Files Part 1
• 5. File formats
  • 5.1. CSV
  • 5.2. JSON
  • 5.3. Markup
• 6. Reading and Writing Files Part 2
• 7. Serialisation/Deserialisation
• 8. OS and File Systems

1. Introduction

The Python standard library io module provides the main facilities for dealing with input and output (IO). Numerous other standard library modules have functionality for dealing with specific aspects and types of IO, including:

fileinput, os, pathlib, contextlib, csv, json, pickle markup, internet, tempfile, and shutil. There are also various third party libraries with useful functionality for read and writing data in specific file formats and processing data on the Web.

2. Standard Input

The builtins module input function reads input from the standard input 'stdin' which is usually the keyboard. This continues until the <ENTER> (or <RETURN>) key is pressed. The function returns a string (without a newline character). A string provided as a parameter is displayed to prompt user input. Run the following at the Python prompt and enter some keyboard input:

n_agents = input("Enter n_agents (a number between 10 and 100) and press the <ENTER> or <RETURN> key:")
print("The input detected is:", n_agents)

3. Streams

The builtins module print function writes to the standard output 'stdout' which is usually the terminal/console/screen.

Stdin and stdout are streams - flows of data. Standard error 'stderr' is also a stream - one where error messages go. Like stdout, stderr is usually written to the screen by default.

The streams stdin and stdout can be redirected to come from a file (in the case of stdin), or go to a file (in the case of stdout). From the Anaconda Prompt the following will stream data from 'stdin.txt' into 'a.py' as it runs:

python a.py < stdin.txt

The following will send output from running 'a.py' into 'stdout.txt':

python a.py > stdout.txt

This would overwrite 'stdout.txt' if it already existed. To append to the end of any existing 'stdout.txt', the following could be used:

python a.py >> stdout.txt

To stream data in and out, the following can be used:

python a.py < stdin.txt > stdout.txt

The stdout of one program can be piped to the stdin of another program using the pipe symbol '|'.

The print

function also includes an option to direct stout to a file. That file has to be open in order for the writing to be successful and the open file should be closed to ensure all the data gets written. Typically reading and writing files uses 'buffers' that read or write a certain number of bytes in one go which is more efficient. Closing a file that is being written to - forces a flush of any partially filled buffer. There is an option in the print function to force a flush which can also sometimes be useful...<

4. Reading and Writing Files Part 1

The following code uses the builtins module open function to open a file in the current directory called 'a.in' and read the file one line at a time streaming the output to the screen before then closing the file:

f = open("a.in")
for line in f:
    print(line)
f.close()

The File Object returned from the above open

function call 'f' is best closed once it is read by calling it's 'close' method as done on line 4 of the code snippet. After this line is executed, 'f' becomes" unusuable. Closing releases system resources and is recommended as good practice altough code will work without doing that.

An alternative where a 'close' method call is not necessary uses the keyword 'with'. The following does effectively the same as the previous snippet:

with open("a.in") as f:
    for line in f:
        print(line)

This saves having to close the file, but it is awkward this way to read several files simultaneously.

The standard library fileinput module helps with reading multiple files simultaneously.

The readlines method

Files are opened for writing in a similar way. Writing to a file can be done as follows:

# Create something to write
a = []
for i in range(10):
    a.append("Coding is fun!");
# Open a file for writing
f = open("a.out", 'w')
# Write a to the file
for line in a:
    f.write(line)
# Close the file
f.close()

Note the additional 'w' parameter passed into the Open Function which prepares the file a.out to be written to. Writing to a file requires different preparation to reading from a file.

There are other optional arguments that can be used with the open function, follow the link for details...

An alternative using the keyword 'with' that does not require closing the file is:

# Create something to write
a = []
for i in range(10):
    a.append("Coding is fun!");
# Write a to file
with open("a.out", 'w') as f:
    # Write a to the file
    for line in a:
        f.write(line)

5. File formats

File formats were briefly introduced in Programming Section 2.2 .

All files are binary files, but some binary files are known as 'text files' - the encoding is generally recognised as text. Text files are typically delimited into lines by a 'newline' code which in Python is '\n'. File formats are usually defined in a file format specification that details the structure of the file. In this section three types of text file formats are described (CSV, JSON and Markup).

GeoJSON example
{
    "type": "FeatureCollection",
    "features": [ {
        "type": "Feature",
        "geometry": {
            "type": "Point",
            "coordinates": [42.0, 21.0]
        },
        "properties": {
            "prop0": "value0"
        }
    }]
}

with open("data.txt") as f:
data = []
for line in f:
    parsed_line = str.split(line,",")
    data_line = []
    for word in parsed_line:
        data_line.append(float(word))
    data.append(data_line)
print(data)

import csv
f = open('data.csv', newline='')
reader = csv.reader(f, quoting=csv.QUOTE_NONNUMERIC)
for row in reader: # A list of rows
    for value in row: # A list of value
        print(value) # Floats
f.close()

import csv
f = open('data.csv', 'w', newline='')
writer = csv.writer(f, delimiter=' ')
for row in data:
    writer.writerow(row) # List of values.
f.close()

import json
f = open('data.json')
data = json.load(f)
f.close()
print(data)

import json
f = open('data.json', 'w')
json.dump(data, f)
f.close()

import os
print(os.environ["PATH"])

print os.listdir(path='.')