SLC S21 Week 6 || Mastering File Handling Part 2: Managing Data Records with Pickle

_{Created by @kouba01 using Canva & PhotoFilter}

Hello Steemians,

In this second part of the competition, we’ll delve deeper into handling data records using Python. Data records often represent structured information such as user details, transaction logs, or inventory items. Unlike plain text files, which are best suited for simple or linear data, data records often benefit from being stored in serialized formats for better structure and efficiency. Python’s pickle module allows us to achieve this seamlessly.

The pickle module is used to serialize and deserialize Python objects, enabling us to store complex data structures (like dictionaries, lists, or custom objects) in files. This provides a practical solution for managing structured data in applications, particularly when combined with graphical interfaces built with Qt Designer.

This week’s challenge will guide you through the following concepts:

• Understanding the serialization and deserialization processes.
• Using the pickle library to store and retrieve structured data.
• Building applications that manage data records, such as a simple address book or an inventory system.
• Incorporating these functionalities into user-friendly GUIs using PyQt5 and Qt Designer.

Why Use Pickle for Data Records?

Using pickle to handle data records offers several benefits:

1. Flexibility: Easily save and load Python objects without converting them into intermediate formats.
2. Efficiency: Directly store complex data structures without writing custom parsers.
3. Interoperability: Share serialized data between programs that use Python.

However, it’s essential to understand that pickle is Python-specific and should only be used for trusted data sources, as it can execute arbitrary code during deserialization.

I. Introduction to Pickle

1. Pickling (Serialization)

Pickling is the process of converting Python objects, such as dictionaries or lists, into a binary format that can be saved to a file. This is especially useful for persisting data across different program runs.

In this example, we define a dictionary object containing a person's details. The open() function is used to create a file named data.pkl in binary write mode (wb). The pickle.dump() method serializes the dictionary into a binary stream and writes it to the file. Once saved, this file can be reopened and used to retrieve the original data.

2. Unpickling (Deserialization)

Unpickling reverses the process by loading the binary data from a file and converting it back into a Python object.

Here, the file data.pkl is opened in binary read mode (rb). The pickle.load() method reads the binary data and reconstructs it into its original form, allowing the program to use the data as if it were never saved.

---

II. Working with Structured Records

1. Saving Structured Data

Structured data often consists of complex objects like a list of dictionaries. For example, we may need to store a list of contact details.

In this example, the contacts list contains dictionaries representing individual records. The list is serialized using pickle.dump() and stored in contacts.pkl.

2. Retrieving Records

Saved records can be retrieved, processed, and displayed, making them reusable across sessions.

The pickle.load() function reads the binary data and reconstructs it into a Python object. The retrieved list is then iterated over, and each contact’s details are printed.

---

III. Advanced Methods with Pickle

1. Adding New Records

Adding new records involves reading the existing data, appending the new data, and saving it back.

Here, the new contact is appended to the existing contacts list, and the updated list is written back to the file.

2. Updating Records

Updating a record requires identifying the specific entry, modifying it, and saving the changes.

The program iterates through the list, finds the contact named "Ali," and updates their phone number.

3. Deleting Records

Deleting a record involves filtering out the unwanted entry and saving the updated list.

---

IV. Integrating Pickle with PyQt5

1. Interface Design

Using Qt Designer, create a simple graphical interface. It includes:

1. Input fields for entering name and phone number.
2. Buttons for actions like adding, viewing, updating, and deleting records.
3. A list widget to display all stored contacts.

2. Linking the Interface

Adding a Contact:

This function retrieves input from the interface, appends the new data to the existing list, and updates the file.

Displaying Contacts:

This function reads the contact list from the file and populates the list widget.

Updating a Contact:

This function allows the user to update the phone number for a specific contact.

---

V. Validation and Error Management

Validating Data

To prevent incomplete input, the program verifies that all fields are filled before proceeding.

Handling Missing Files

When the file is not found, the program initializes an empty list to handle the absence of data gracefully.

---

Conclusion

These examples showcase how to effectively manage structured data using the Pickle library and integrate these functionalities into a PyQt5 interface for enhanced user experience. The detailed step-by-step process ensures a comprehensive understanding of the topic, enabling you to build robust applications for real-world use cases.

Homework Tasks

Task 1 (3.5 points):

This task involves creating an application focusing on tracking and managing outdoor exercises like running, walking, and cycling. The application provides the following functionalities:

1. Adding a new type of exercise.
2. Logging new exercises with details such as date, duration, and distance.
3. Displaying the exercise history for a selected type, along with an overview of total duration, number of exercises, and total energy burned.

Key Requirements:

• Use a graphical interface containing:
  • A section to add new exercises, including input fields for type, date, duration, and distance, as well as a button to log the exercise.
  • A section to display the exercise history, with options to filter by type, a table for detailed records, and a summary of totals.
• Populate the exercise type dropdown dynamically from a text file (typesCourse.txt).
• Store new exercises as records in a binary file (Exercice.dat) and compute energy burned using the formula:
  Energy = Distance (km) × Weight (kg) (assuming a fixed weight of 85 kg).
• Automatically update dropdowns and the text file when new types are added.
• Validate input fields and display error messages for invalid entries or empty fields.
• Provide summaries and totals, such as cumulative duration, energy, and exercise count, in a dedicated section of the interface.

Implementation Guidelines:

• Complete the provided graphical interface (course.ui) by assigning meaningful names to its components.
• Develop modular functions to:
  • Populate dropdown lists from the text file.
  • Add new exercises to the binary file, update dropdowns, and reset input fields after submission.
  • Retrieve and display exercise data based on user-selected filters, and compute totals for the summary table.
• Ensure all resources are used from the specified Ressources directory, and save solutions in a dedicated work folder.

Icon resource:

Task 2 (3.5 points):

We aim to develop an application for managing a TV channel database stored in the file listChannels.dat. Each channel is described by the following attributes:

• Channel Name: A non-empty string.
• Frequency: An integer in the range [10,000, 20,000].
• Polarization: A character (V for Vertical or H for Horizontal).
• SR FAC: A five-digit integer.
• Cryptage: A non-empty string.
• Category: A non-empty string.

Application Features:

1. Search Functionality:
   Users can search for channels by name, part of a name, or category. The results are displayed in a TableWidget when the "Find" button is clicked.

2. Dynamic Category Management:
   The list of categories is stored in the file categories.txt. When the application is launched, these categories are automatically populated into two dropdown menus (ComboBoxes).

3. Add a New Channel:
   Users can add a new channel to the file listChannels.dat using a dedicated form. The form includes fields for name, frequency, polarization, SR FAC, encryption, and category. Validation is performed to ensure all fields are correctly filled.

4. Clear Form Fields:
   The "Cancel" button clears all the fields in the "Add New Channel" form.

---

Practical Requirements:

Design the graphical interface with the following components:

• Search GroupBox:

  • A label with the text "Name".
  • A text input field for entering the name of a TV channel.
  • A dropdown menu for selecting a category.
  • A button labeled "Search" to perform searches in listChannels.dat.

• Channel List GroupBox:

  • A TableWidget with columns for channel attributes: "Channel Name," "Frequency," "Polarization," "SR FAC," "Encryption," and "Category".

• New Channel GroupBox:

  • A form with input fields for channel attributes and buttons to add or cancel operations.

1. Develop Modular Functions in channel.py:
   • filList Function: Populates the dropdown menus with categories from the file categories.txt.
   • Add Function: Adds a new channel to listeChannel.dat after validating all fields.
   • Find Function: Searches for channels based on user input (name or category) and displays the results in the TableWidget.

Task 3 (3 points):

The application enables users to input, validate, save, and display data in a user-friendly manner using Python and PyQt5. The data is stored in a binary file and displayed in a table for easy access and review.

---

Application Features

1. Form Input:

   • The form collects developer details including:
     • Civility
     • Name (non-empty).
     • Email (must include @).
     • First language used (selected from three radio buttons).
     • Favorite programming languages (multiple selections via checkboxes).

2. Validation:

   • The application validates the name and email fields to ensure they are correctly filled before saving the data.

3. Data Storage:

   • Developer information is stored as a record in a binary file named sheet.dat.

4. Data Display:

   • Saved records are retrieved from the binary file and displayed in a TableWidget with the following columns:
     • Civ
     • Name
     • Email
     • First language used
     • Programming languages

5. Reset Form:

   • A "Clear" button resets all form fields to their default states.

6. Exit Application:

   • A "Close" button closes the application.

---

Technical Requirements

1. Interface Design:

   • The graphical interface (Developer_Sheet.ui) must include:
     • Input fields for name and email.
     • Three radio buttons for civility selection.
     • Six checkboxes for programming language preferences.
     • A dropdown menu for selecting the first language used.
     • A TableWidget for displaying saved records.
     • Buttons for "Save Record," "Clear," and "Close."

2. Functionality:

   • Develop modular functions for:
     • Processing and validating form input.
     • Saving records to the binary file.
     • Loading and displaying saved records in the table.
     • Resetting form fields.
     • Closing the application.

---

Expected Output

When the user inputs valid information and clicks the "Save Record" button:

• The information is saved in the binary file sheet.dat.
• All records in the file are displayed in the table, including the newly added entry.
• A confirmation message is displayed indicating successful operation.

If any field is invalid, an error message is shown, and the data is not saved.

Resource image:

Contest Guidelines

• Write your posts in any community or your personal blog.
• Use the title: Mastering File Handling Part 2: Managing Data Records with Pickle.
• Include only #steemexclusive content and copyright-free images (with sources).
• Submission schedule: December 2nd, 2024, 00:00 UTC to December 8th, 2024, 23:59 UTC.
• Use these tags: #dynamicdevs-s21w6, #python, #qt, #pickle, #teaching, #tunisia, and #steemexclusive.
• Post the link to your entry in the contest comments.
• Invite 3 friends to participate.
• Share your post on Twitter and link it in your post.
• Plagiarism or AI-generated content is prohibited.

Rewards

At the end of the week:

• Four top-quality posts will be selected for additional rewards.
• SC01/SC02 will upvote outstanding posts (not guaranteed for all entries).

We encourage you to explore Python’s serialization capabilities through the pickle module and demonstrate your creativity by building applications that manage structured data records. Happy coding!

Best Regards,
Dynamic Devs Team