From boris at kolpackov.net Wed Sep 1 10:40:48 2004 From: boris at kolpackov.net (Boris Kolpackov) Date: Wed Sep 1 10:35:35 2004 Subject: make: custom up-to-date Message-ID: <20040901154048.GA7683@kolpackov.net> Good day, GNU make has only one way of determining up-to-date-ness: modification timestamp. This is an efficient and in most cases sufficient mechanism. But every now and then a situation arises where you want a custom up-to-date check. If such a need is local and you have full control over the makefile fragment then everything can be arranged using so-called "sentinel" file technique. The technique is based on the way make decides when to update targets and whether the targets were actually updated. Consider a small example: foo: foo.o $(CC) -o $@ $^ foo.o: foo.c $(CC) -o $@ -c $< Let's assume both `foo.o' and `foo.c' exist and `foo.c' is older than `foo.o'. After make compares timestamps of `foo.c' and `foo.o' it realizes that `foo.o' should be updated and runs the corresponding command. After the command is successfully completed, make compares modification times of `foo.o' before and after the command execution. If they are the same, make concludes that `foo.o' was up-to-date and there is no need to rebuild anything that depends on it (`foo' in our case). The "sentinel" file technique is based on this before-after comparison. Suppose `foo.c' gets updated frequently but the content of the file often does not change. It would be nice to use custom up-to-date-ness check on this file which will trigger updates only if content of the file has changed. One way to achieve this is to use `foo.c.md5' "sentinel" file: foo: foo.o $(CC) -o $@ $^ foo.o: foo.c.md5 $(CC) -o $@ -c $(<:.md5=) foo.c.md5: foo.c @md5sum $< | cmp -s $@ -; if test $$? -ne 0; then md5sum $< > $@; fi In this example `foo.c.md5' serves two purposes: it keeps the previous md5 hash sum and triggers an update of `foo.o' when the previous and current hash sums do not match. While this example may look neat, it has one problem: it is not user-makefile-transparent. To understand what I mean let's transform our original example above to be a little closer to real life: # Things we cannot change. # foo: foo.o foo.o: foo.c foo.h # Things we can change. # %: %.o $(CC) -o $@ $^ %.o: %.c $(CC) -o $@ -c $< To implement md5-based up-to-date check we need to change this line foo.o: foo.c foo.h to be foo.o: foo.c.md5 foo.h.md5 This is not very practical since it will most likely lead to changes in a lot of user makefiles (even if it is acceptable, sure it would be nice not to have to). It is also bad because we embed a particular method into user makefiles which makes the whole construction very inflexible. The ideal solution would require changes only to the part marked "Things we can change" which is usually located in a common file included by every user makefile. A long story short you can't do it in current GNU make. But we are just a few logical features away from being able to. Right now pattern rules allow you to add prerequisites to the original rule. What's missing is the ability to remove and query prerequisites from the original rule. If we had this ability we would be able to automatically mangle `foo.c foo.h' to become `foo.c.md5 foo.h.md5'. In fact I went ahead and implemented those two features in `bk' patch-set ( http://kolpackov.net/projects/make/bk/ ). Here's how it works: # Things we cannot change. # foo: foo.o foo.o: foo.c foo.h # Things we can change. # %: %.o $(CC) -o $@ $^ %.o: %.c.md5 $$(addsuffix\ .md5,$$^) $$- $(CC) -o $@ -c $(<:.md5=) .PRECIOUS: %.md5 %.md5: % @md5sum $< | cmp -s $@ -; if test $$? -ne 0; then md5sum $< > $@; fi The second pattern rule is where all the action happens. The first prerequisite pattern (`%.c.md5') ensures that we have a `.c' file for every `.o' file. The second prerequisite (`$$(addsuffix\ .md5,$$^)') is where changing `foo.c foo.h' to `foo.c.md5 foo.h.md5' is done. `$$^' expands to the list of prerequisites from the original rule (just like in command script). Note, that we need to escape it so that it will survive the first round of expansions (happens when makefile is read). And the last prerequisite (`$$-') is a special variable that expands to nothing but also removes all the prerequisites that came from the original rule (`foo.c foo.h' in our case). The complete example shown above is available in the `bk5' patch-set. If you have made it this far, thank you for your time. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 652 bytes Desc: Digital signature Url : http://www.kolpackov.net/pipermail/notes/attachments/20040901/939d6ee2/attachment.bin From ian_dunbar at hotmail.com Thu Sep 9 10:35:03 2004 From: ian_dunbar at hotmail.com (Ian Dunbar) Date: Thu Sep 9 10:29:30 2004 Subject: make: custom up-to-date In-Reply-To: <20040901154048.GA7683@kolpackov.net> References: <20040901154048.GA7683@kolpackov.net> Message-ID: <20040909153503.GA11539@kolpackov.net> Hi Boris, On your custom up-to-date system there seems to be one small bug/feature. If everything is up to date according to the md5 test, but changed according to timestamps you can end up with this kind of situation: >make md5sum a.c | cmp -s a.c.md5 -; if test $? -ne 0; then md5sum a.c > a.c.md5; fi md5sum b.c | cmp -s b.c.md5 -; if test $? -ne 0; then md5sum b.c > b.c.md5; fi etc.... i.e. On each make invocation, md5sum is always run for every .c file that is "touched" but not "changed". Also you don't get the reassuring "make: `foo' is up to date." message. One possible workaround might be, in the md5 rule, to touch the .c to be the same (older) date as the .md5 as an else part of your if statement. Unfortunately touch doesn't have a use-same-timestamp-as-this-other-file option, so getting the timestamp information might be tricky. Best regards, Ian From boris at kolpackov.net Thu Sep 9 10:40:39 2004 From: boris at kolpackov.net (Boris Kolpackov) Date: Thu Sep 9 10:32:12 2004 Subject: make: custom up-to-date In-Reply-To: <20040909153503.GA11539@kolpackov.net> References: <20040901154048.GA7683@kolpackov.net> <20040909153503.GA11539@kolpackov.net> Message-ID: <20040909154039.GA11575@kolpackov.net> Ian Dunbar writes: > On your custom up-to-date system there seems to be one small bug/feature. > > If everything is up to date according to the md5 test, but changed > according to timestamps you can end up with this kind of situation: > > >make > md5sum a.c | cmp -s a.c.md5 -; if test $? -ne 0; then md5sum a.c > a.c.md5; > fi > md5sum b.c | cmp -s b.c.md5 -; if test $? -ne 0; then md5sum b.c > b.c.md5; > fi > etc.... > > i.e. On each make invocation, md5sum is always run for every .c file that > is "touched" but not "changed". Well, that's your custom up-to-date check therefore it is run every time the up-to-date-ness is in question. > Also you don't get the reassuring "make: `foo' is up to date." message. That's actually a make bug. > One possible workaround might be, in the md5 rule, to touch the .c to be > the same (older) date as the .md5 as an else part of your if statement. > Unfortunately touch doesn't have a use-same-timestamp-as-this-other-file > option, so getting the timestamp information might be tricky. Actually it does: see -r option to POSIX touch. I just rewrote the command above like this: md5sum $< | cmp -s $@ -; if test $$? -ne 0; then md5sum $< > $@; \ else touch -r $@ $<; fi This is a nice optimization but we should be careful here: we are changing timestamp of a file behind make's back which can back fire (e.g., if we have more than one target built from a particular prerequisite, one via md5 and the other directly). -boris -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 652 bytes Desc: Digital signature Url : http://www.kolpackov.net/pipermail/notes/attachments/20040909/23891264/attachment.bin