48. Handling Large Files with io Module

1. Opening a File with Buffered Reading (io.open)

Reading a file in buffered mode to handle large files efficiently.

Copy

import io

with io.open('large_file.txt', 'r', buffering=1024) as file:
    data = file.read(1024)  # Read 1 KB of data at a time
    print(data)

This reads the file in chunks of 1024 bytes, which helps reduce memory usage when dealing with large files.

---

2. Reading a Large File Line by Line

Reading a file line by line to avoid loading the entire file into memory at once.

Copy

import io

with io.open('large_file.txt', 'r', buffering=2048) as file:
    for line in file:
        print(line.strip())

This reads the file line by line, which is memory-efficient for large files.

---

3. Writing Large Data to a File Using Buffered Writing

Writing large amounts of data to a file using buffered I/O.

Copy

import io

data = 'A' * 10**6  # 1 million characters

with io.open('large_output.txt', 'w', buffering=2048) as file:
    file.write(data)

This writes large chunks of data to the file, and buffered I/O optimizes the writing process.

---

4. Reading a Binary File Efficiently

Using buffered reading for binary files (e.g., images, videos).

Copy

import io

with io.open('large_image.png', 'rb', buffering=1024) as file:
    chunk = file.read(1024)
    while chunk:
        print(chunk)
        chunk = file.read(1024)

This reads a binary file in chunks, which is efficient for handling large binary files.

---

5. Reading a Large File in Chunks Using io.BufferedReader

Using BufferedReader for efficient reading of large files.

Copy

import io

with io.open('large_file.txt', 'rb') as file:
    buffered_reader = io.BufferedReader(file)
    chunk = buffered_reader.read(2048)
    while chunk:
        print(chunk)
        chunk = buffered_reader.read(2048)

This uses BufferedReader to read large files in smaller, manageable chunks.

---

6. Writing Binary Data with io.BufferedWriter

Writing binary data efficiently with buffered writing.

Copy

import io

data = b'Hello, this is some binary data.'

with io.open('output_file.bin', 'wb') as file:
    buffered_writer = io.BufferedWriter(file)
    buffered_writer.write(data)

Buffered writing helps efficiently manage large binary writes to a file.

---

7. Using io.FileIO for Low-Level File Handling

Low-level file operations with buffered I/O for large files.

Copy

import io

with io.FileIO('large_file.txt', 'r') as file:
    buffered_reader = io.BufferedReader(file)
    chunk = buffered_reader.read(4096)
    while chunk:
        print(chunk)
        chunk = buffered_reader.read(4096)

io.FileIO provides a low-level interface for file handling with buffered I/O, ideal for large files.

---

8. Seeking and Reading Large Files Using io.Seek

Efficiently seeking and reading from a large file using the seek method.

Copy

import io

with io.open('large_file.txt', 'r', buffering=1024) as file:
    file.seek(5000)  # Move to byte 5000
    data = file.read(1024)
    print(data)

This moves to a specific byte position in the file and reads a chunk, optimizing file reading.

---

9. Writing Large Data to a Binary File Using io.FileIO

Efficiently writing large binary data using FileIO with buffered I/O.

Copy

import io

data = b'\x01\x02\x03\x04' * 10**6  # Large binary data

with io.FileIO('output_file.bin', 'wb') as file:
    buffered_writer = io.BufferedWriter(file)
    buffered_writer.write(data)

This demonstrates buffered writing for large binary data, reducing memory usage.

---

10. Copying Large Files Using io.Copy

Efficiently copying large files using buffered I/O.

Copy

import io
import shutil

with io.open('large_file.txt', 'rb') as source:
    with io.open('copied_large_file.txt', 'wb') as destination:
        shutil.copyfileobj(source, destination, length=1024)

Using shutil.copyfileobj with buffered I/O allows efficient copying of large files in chunks.

---

These examples show how the io module provides powerful tools for efficiently handling large files, whether you're working with text or binary data. By using buffered reading and writing, you can significantly reduce memory usage and improve performance in file-intensive operations.