1

My question is very simple; however, I am struggling with it to automate a few folder copy stuffs. Can anyone give me a quick command which will let me find out a directory created today and compress it?. Something like this. # find /u04/CSPRD/pdf/* -mtime -1 -exec tar -cf copy.tar {} ; The above command doesn’t give me the desired output as all the sub directories within the directory are not being compressed. PS: the root folder which is created today is a huge tree structure with huge number of sub- directories and files within it.

2

Windows systems have zip which does compress a whole directory. You can get
zip for Linux but it is not very good.

The standard Linux way to do what you ask is already in the posts above.

#… At the sending end.

cd /u04/CSPRD/pdf
tar cf - 20171017 | gzip > xfer_20171017.tgz

#… Transfer the single file that was created. (It is not a hieracrchy –
it is a set if directions for creating a hierarchy.)

sftp (or whatever) target DC1, file xfer_20171017.tgz

#… At DC1, unpack the file.

cd /u04/CSPRD/pdf
gunzip < xfer_20171017.tgz | tar xv -

Look up gunzip. It is usually a link to gzip which makes it decompress, but
on some systems you have to use an explicit option like gzip -d.

Your complete data tree will be extracted to re-create exactly the
hierarchy you had before, with a couple of exceptions:

… The file owners and groups are transferred as numeric values. If DC1 and
DC2 have different ids for the same users, the owners may appear different
or appear as numbers.

… Sparse files (with whole blocks of zeros) will be unloaded using real
blocks for the original gaps.

3

You’re welcome.
Nevertheless, search in ITtoolbox, we had some threads about tar vs cp, as usual with excellent comments from Paul and others.
I really think you should tar cf - $DAY | (cd /mnt | tar xf - )

Franck

4

I now feel embarrassed… I was thinking too much and missed a simple one which is just in front of me… :slight_smile: Thank you Mr. Franck Cornaz. You opened my eyes. I did mount an NFS share using my management IP and it was very simple.

I just created a very simple script to copy the directory. As I did an NFS mount, I don’t require tar/compress anymore. Just do a direct cp to the destination path as below:

#!/bin/bash
DAY=$(date +“%Y%m%d”)
cd /u04/CSPRD/pdf/ && cp -rp $DAY /cs_font/Ftp_Data/Consolidated_Statement_PRD/pdf/

Now, I will do the remaining modifications wherever necessary and create a job.

Thank you all for the help and appreciated every one. :slight_smile:

5

If both machines are in the same network, use rsync. Gets only the differences and will update permissions and ownership if they change on the source.

6

Can you do a nfs mount between yout two servers? so no need to ftp or scp
Suppose you mount the place where you want to copy on /mnt

DAY=$(date +“%Y%m%d”)
$ cd /u04/CSPRD/pdf/$DAY
$ mkdir -p /mnt/$DAY # in case it doesn’t exist
$ tar cf - $(find . -type f -mtime -1) | (cd /mnt/$DAY ; tar xvf -)

Could be the same with backup and restore and you keep your NFS4 ACL and sparse files (if any) will still be sparse.

7

I think we are out of track. Apologies if my actual request was confusing as I see a lot of comments including someone asked my why I need to tar my root folder!. By the way, root folder doesn’t mean /root for me. What I meant is the parent directory. Anyway, Let me re-instate my scenario.

I have 2 AIX 6.1 servers (planning to upgrade to 7.1 soon) - DC1 and DC2. There is an application which generates a directory on DC2 inside /u04/CSPRD/pdf/ on specific dates ; say 1st, 2nd, 4th, 6th, 10th and 15th of every month. The folder generated will be in this format {date +“%Y%m%d”}.
Now, I need this particular folder to be copied to DC1 server through a script. A particular tool will call this script to do the copy. Here is the catch!. This tool cannot copy a directory, only file copy is possible. Hence, I need a script to compress the whole directory to a single file so that my third party tool could facilitate file copy to the destination server DC1.

The simplest solution I thought was, create a simple scp script after creating a password less ssh option between DC1 and DC2. However, this is not possible on my servers as the permission set for the parent folders (/u04) on DC1 and DC2 are 777 (there are some application reason behind it).

Next solution was to do ftp. However, only file copy is possible through ftp.

Now, I have a simple script ready to do ftp without password for which I need the whole directory to be archived/zipped first.

So my question is very simple. Is there any way to compress the whole directory /u04/CSPRD/pdf/‘date +“%Y%m%d”’ using a script?. As the directories are created only on specific dates, I prefered creating a script as below;however, it doesnt compress all the files/directories.

find /u04/CSPRD/pdf/* -mtime -1 -exec tar -cf copy.tar {} ;

FYI, this is who my tree structure looks like:
bash-4.2# pwd
/u04/CSPRD/pdf
bash-4.2# ls -ld 20171017
drwxr-xr-x 5 auhbknet dba 256 Oct 17 12:37 20171017
bash-4.2# ls -ld 20171015
drwxr-xr-x 5 auhbknet dba 256 Oct 16 02:45 20171015
bash-4.2# cd 20171015
bash-4.2# ls -lrt
total 0
drwxr-xr-x 3 auhbknet dba 256 Oct 16 02:44 email
drwxr-xr-x 3 auhbknet dba 256 Oct 16 02:44 Paper
drwxr-xr-x 3 auhbknet dba 256 Oct 16 02:45 eStatement

Inside these directories, I have more and more sub-directories and files.
Let me know if you need more clarity on my requirement :slight_smile:

8

In AIX (tested on AIX6.1 and 7.1) giving no files doesn’t produce the funny msg, but a syntax error (classical usage). This is the reason why I put the ladate file in the list of files to archive.

The -L /dev/null is OK but not as written above: “tar cf /tmp/myArchive.tar -L /dev/null”
It should be either X/OPen version with minus “tar -cf /tmp/myArchive.tar -L /dev/null” or
Berkeley options (no minus) “tar cfL /tmp/myArchive.tar /dev/null”.
In both cases the output is empty AND there is no /tmp/myArchive.tar file created.
Luckily, gzip creates a 20 bytes output (as described above) with

tar cfL - /dev/null | gzip > /tmp/xxx.tar.gz
ls -l /tmp/xxx.tar.gz
-rw-r–r-- 1 franck staff 20 Oct 18 14:22 /tmp/xxx.tar.tgz

Franck

9

Found a way to bypass the “no files” check on another forum, but I can’t test whether it works on AIX. You might want an empty archive just to record the fact that you ran every day, for example.

The example I saw was to specify an InputList with the -L option, and have the list empty. It satisfies the args checking, but makes an empty archive. (Note it is not archiving /dev/null, it is using /dev/null as the source of the file name list.)

tar cf myArchive.tar -L /dev/null

Seems to me if the required files are in a namelist, which sometimes happens to be empty, it would still work.

tar cf - -L ./ArchList.txt | gzip > myArchive.tgz

Don’t be surprised that the .tgz is not empty. First, the “empty” tar archive is padded to 10240 bytes to be compatible with tape blocking requirements. Second, even for an empty file, gzip records a non-empty initial decode table, plus the original file name and size.

10

Franck,

Tar has a joke message if you don’t specify any files. It says:

“Cowardly refusing to create an empty archive”.

John,

My last serious encounter with AIX started in 2004, and it was to support an end-of-life critical system that had been locked down for seven years. So it was using an AIX from last century – 4.3 I think.

All the diagnostics I wrote had to run at every node on a mixed system, with about 140 at Solaris 2.6 and 24 at AIX 4.3. I got very picky about which options to use, even on dumb stuff like ping, ps, and tail. I still feel the pain.

11

I think my scripts were written around AIX 5.1, maybe 4.3.3? They work, so I leave them alone. About 10 years ago I started testing my scripts on Linux just in case our company did an OS/platform move. We moved to Linux on Intel 2 years ago, and the scripts work there as well with no changes.

12

AIX has the mmin, cmin and amin since AIX6.1 (and I think AIX5.3 TL6, that means 10 years ago).
I agree with Dan and Paul, just a touch on a directory will save the whole tree.
But it is possible to say ‘find . -type f -mtime -1’ to restrict to files and not directories.
Like John, I will do some ‘touch -m -t $(date +"%Y%m%d0000.00) ladate’ and a find -newer ladate.
But, if no change occurs during the day, my proposed syntax is wrong, since tar would have no file to save (syntax error, I guess). so put the “ladate” file in your directory, and save it
So maybe we create a first time file “ladate” in the current directory.

cd /path/to/dir

the ladate file will be saved whatever happens, maybe the only file archived if no change during the day.

touch /tmp/ladate.future # just before we archive
tar cf - ladate $(find . -type f -newer ladate) | gzip >/tmp/xxx.tar.gz
mv /tmp/ladate.future ./ladate # tomorrow we will refer to this file

13

I still think that the find will produce a list of all directories and
files, and be essentially random in respect of directories because the
timestamp check is affected by other actions. You will get:

./myDir1
./myDir1/F1.pdf
./myDir1/F2.pdf
./myDir2
./myDir2/F3.pdf
./myDir2/F4.pdf

Tar will see ./myDir1 and dump that subtree (including the files in it),
and then see F1 and F2 and dump them for a second time. And the deeper the
tree, the more duplicates you get – one set of duplicates for every
directory level.

Also, if there are a huge number of files, the command line to tar will be
too long for the limit on args passed to a process. If that couldn’t
happen, we wouldn’t need to have written xargs.

But xargs can’t fix this either. The point is that xargs runs as many of
the commands as needed to accomodate all the args in the list. That’s fine
for commands like mv or rm. But for tar, every command run by xargs will
rewrite your .gz file from scratch, losing previous instances.

Probably to do this for specific files, you need to make a list of just the
files (not directories) and pass that list to tar:

DATE=“$( date ‘+%Y%m%d’ )”
ARCH=“/home/myArchStore/yourArchive.${DATE}.gz”
LIST=“/tmp/tarPdf_${DATE}.$$.txt”

find . -type f -mtime -1 > “${LIST}”
echo 1>&2 “Archiving $( wc -l < “${LIST}” ) files to ${ARCH}”
tar cf - -L “${LIST}” | gzip > “${ARCH}”
rm -f “${LIST}”

But in general, organising your recurrent data by hour, day, month, year
(depending on volume, frequency and your specific housekeeping regime) is
always the smart way to go.

Incidentally, if you put your archive into your PDF directory, then after a
couple of days you will start archiving the archives into other archives
recursively (because the find will list them), which will snowball
astonishing quickly.

Like I said before, you need a coherent design for a housekeeping strategy
before you can do anything sensible with this. For example, have you
considered how you recover specified files from archives – how do you even
know which archive to search on?

14

Find on AIX is to coarse to use the mtime option.

The way I got around AIX not having these features was to use the touch command and create a file with the exact date/time stamp that I wanted. Then use the -newer option specifying the touch file just created in the find command.

-mtime -1 gets everything that has a time stamp 24 hours or newer. This would not work for what I needed. At the beginning of the backup script, I would create the temporary touch file with the time needed, an unknown number of files would be created in a central directory, find would then scan the directory using the touch file to find everything created since the start of the backup script.

15

My bad memory. -mtime -1 does mean less than one day ago, +1 means more
than one day ago.

But it is still junk. I can’t even understand the manual here, but for
times it says:

“File was last accessed n*24 hours ago. When find figures out how many
24-hour periods ago the file was last accessed, any fractional part is
ignored, so to match -atime +1, a file has to have been accessed at least
two days ago.”

I think “more than” means strictly > and not >=, so it needs a whole extra
24 hours to trigger. Personally, where it matters to me, I tend to stat
(istat in AIX) the file and get the seconds, and compare it with the
seconds returned now from date ‘+%s’. If you have seconds, you can figure
your own days arithmetically.

Also, +1 is not yesterday. It counts hours. So if you run your find -mtime
+1 at 15:30 on Tuesday, it selects files that were last updated before
15:30 on Monday. Part of yesterday is in and part is out.

… Linux has (but AIX does NOT) a -mmin option that counts in Minutes.

… Linux has (but AIX does NOT) a -daystart option that counts time from
00:00 of Today, rather than from current wall time. So it does count
distinct days without overlap.

It would be a whole lot easier for everybody if your files showed up in
directories with names like:

/u04/CSPRD/pdf/2017_10_17/whatever

or you could just make the directories and have /u04/CSPRD/pdf be a link to
today’s, updated by a cron job at midnight.

Then you could just compress the last directory but two, archive the last
directory but seven, and remove the last directory but 30. You don’t have
to worry about dates, whole days, finding stuff and all that junk.

And as Dan notes, directories get a modified timestamp when they are
themselves modified. If you have a tree, then creating a file only modifies
the immediate directory. And removing a file also marks the tree as
modified, even though every file left in it is now much older than the
directory itself.

Files and directories can also be touched – a fake modification – and the
timestamp can be set to any date you want, past or future. File system
timestamps are completely useless and unreliable except in carefully
regulated areas (like source repositories and release systems).

16

Some more after Paul’s comments:

  • do you have any reason to use tar and not backup since you’re on AIX? That means, do you have to restore on other OS (Linux, Solaris, etc.)
  • Which AIX version do you use (oslevel -s)
  • the current implementation of tar in AIX7.1 and 7.2 allows 2^64 bytes archive (infinite?) on JFS2
  • I’m sure it was 8 Go only a log while ago (AIX5.2 ?)
  • if it’s really big, you should modify the ulimit (Paul wrote it).
  • AIX tar does not compress (gnutar has a z option (tar cz)
  • why do you want to compress each file and not the whole archive?
  • do you have any reason to store in absolute path (find /xxx/yyy/zzz/*). You could do (cd /xxx/yyy/zzz && tar)
  • If you don’t have too many files less than 24 hours old, you could for instance write

cd /xxx/yyy/zzz &&
tar cf - $(find . -mtime -1) | gzip > yourArchive.$(date +“%Y%m%d”).gz

find will produce the list of files less than 1 day old, tar will save all these files on its standard output, which is gzip input.
You can use compress or pack if you don’t want gzip.
the line length in AIX is 1 Mega byte long, so if you filenames are something like 32 characters long, you can find recent 32000 files.

That’s an idea.

17

Re: "PS: the root folder which is created today is a huge tree structure with huge number of sub-
directories and files within it. "

Your root directory was not created today. It was created at installation. The mtime and ctime of the root directory have been updated.

Never backup based on a directory time (your “find” command does not do so). Many files in a directory may have been updated, with no effect on the directory’s timestamps.

I think, but I"m not sure, that " -mtime -1 " will work as you intend, but I don’t know much about AIX. " -mtime +1 " would find file modified more than 1 day ago. " -mtime -1 " would find files modifed within 1 day (today).
See

18

Your original command idea is too simple – in fact, it is extremely wrong.

(1) find is recursive. It will list every directory, subdirectory and file
below …/pdf. So it will pass that huge list of names to -exec. So it will
run tar for the top directories, and then again for every subdirectory, and
then again every file separately, all the way down the tree. So if you have
5000 files in your “huge tree structure”, it will keep writing the same
file copy.tar over and over, each time with a bunch of files. And in fact,
the last file it writes will only contain the last few files that xargs
passed to it (because xargs splits the file list into multiple commands if
it is too large for a single command line).

(2) -mtime -1 just means “older than 1 day”. So the second day you run it,
it will write out yesterday’s data, and then it will write out the day
before all over again. And also, both those will go to copy.tar, so you
will lost one or the other of the days. On the seventh day, it will archive
or compress all seven days all over again. And so on forever.

You need some way to only do each day’s data once, like renaming each day’s
pdf so it includes the date, so it does not match the find parameters after
the first time.

The same kind of thing will happen if you use find with xargs and gzip.
gzip is smart enough to avoid compressing files that are already
compressed. It will just give you 5000 diagnostics like “Skipped filename
– appears to be already compressed”. That’s on the first day – by the end
of the week you will get 35000 messages and counting.

Experience says you will run out of space even for compressed archives in a
short while. You need to set out a coherent strategy for dealing with
archives that will be good in a couple of years time. You just can’t keep
huge archives for long periods, unless you build your own cloud. Do some
math and find out when you get into the petabyte range. It may be sooner
than you think.

19

Just saw you are on AIX. You may have a restricted choice of compression
tools, unless you download something.

I see in other forums that AIX gzip may not handle huge files, and also
that the ulimit for file size may need increasing for your user (these are
two separate problems).

You also seem to be in a path that says /pdf/, so you might want to
hand-compress a few files and discover how well they shrink.

20

I don’t understand the requirement.

Tar creates an archive of the directory and all its subtree. It has a
facility to compress the archive. It does not remove any files, so it uses
extra space and holds both the archive, and the original data and directory
structure. If you just want an archive but still need the original data,
that is what you get. (Originally, it was a Tape ARchive so you could keep
in it a fire safe or offsite).

Tar does not compress the files. It makes an archive and then compresses
the whole thing in one go, headers, directories and files as a blob. So
anytime you want a single file back, you have to ask tar to decompress the
whole thing and spit out the one file you want.

If you want to compress all the files but keep the directory tree and have
the files online, then you need to run a compression tool, and have it
recurse down the directory structure. There are various tools that save
slightly more space than others but cost a lot more CPU. I would try gzip
which is standard and effective. It also recurses down a whole directory
tree (use the -r option). However, any time you want to access a file, you
have to unzip it again (temporarily or permanently).

The space you save in compression is variable. Text files and code compress
well. Most stuff that is already encoded in some way (jpg, mp3, pdf) saves
almost no space because it is already reduced, and it is also much slower
(because the search for shorter encoding has to search deeper).