Finding missing data in a table for a certain period of time with PostgreSQL

in missing-data •  7 years ago 

At my work, there is a system that inserts data into the DB​ hourly. From time to time some data is missing randomly, for example, 2 hours of a day in January, 10 hours in February, 24 hours in May, etc. So I need a SQL query that shows me the missing hours so that I can back-fill the missing data later. Since I know that data must be filled hourly, I use the following query to achieve my goal, in particular, to find the hours of dates from 2017 to 2018 when data were missing:

\copy (select missing_date
from generate_series('2017-01-01'::timestamp, '2018-01-01'::timestamp, '1 hour'::interval) missing_date
where to_char(missing_date, 'YYYY-MM-DD HH24') not in (select to_char(dt, 'YYYY-MM-DD HH24') from my_table_name)
order by missing_date) to '/tmp/missing_date.csv' with csv delimiter ',';

Now let's break the query into parts.

\copy (select *** from ***) to 'path_to_file' with csv delimiter ','

to write the query result into a csv file using ',' as the delimiter.

generate_series(start_datetime, end_datetime, interval) [as] column_name

to gene​rate a datetime sequence using the interval.

Since the type of the 'dt' column is timestamp and data are filled not exactly at the beginning of an hour, I use to_char(dt, datetime_format) to convert the timestamp to a proper format.

The query might seem to be obvious for SQL experts, but I think it might be useful for those guys who are to PostgreSQL.

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Congratulations @nmd! You have received a personal award!

1 Year on Steemit
Click on the badge to view your Board of Honor.

Do not miss the last post from @steemitboard:
SteemitBoard and the Veterans on Steemit - The First Community Badge.

Do you like SteemitBoard's project? Then Vote for its witness and get one more award!

Congratulations @nmd! You received a personal award!

Happy Birthday! - You are on the Steem blockchain for 2 years!

You can view your badges on your Steem Board and compare to others on the Steem Ranking

Vote for @Steemitboard as a witness to get one more award and increased upvotes!