How do you use flock and clean up lock files?

So I'm using flock, but I have this noxious race condition when I try to
clean up the lock files.

I need a way of knowing if anything has a file open at the same time as
me, or I must always leave lock files lying around. Any suggestions?

Here is the sequence..
Process A: fd = open( "lockfile", 'a')
Process A: flock(fd, LOCK_EX)

Process A: do stuff in critical section...

Process B : fd = open( "lockfile", 'a')
Process B: flock(fd, LOCK_EX) // Blocks waiting for lock

Process A: Finishes, doesn't know about B, wants to clean up...
Process A: unlink fd
Process A: close(fd)

Process B: Unblocks, B grabs lock.

Process C : fd = open( "lockfile", 'a') // CREATES A NEW ONE!
Process C: flock(fd, LOCK_EX) // DOESN'T BLOCK!

Here is a chunk of ruby that demonstrates this...

fork do
   fd = open( "lockfile", 'a')
   puts fd.stat.ino
   fd.flock(File::LOCK_EX)
   puts "Sleeping"
   sleep 10
   File.unlink "lockfile"
   fd.close
end

sleep 1
fork do
   fd = open( "lockfile", 'a')
   puts fd.stat.ino
   puts "Waiting for lock"
   fd.flock(File::LOCK_EX) # Blocks waiting for lock
   puts "Got lock"
   sleep 100
end

sleep 15

puts "Open file #{Time.now}"
fd = open( "lockfile", 'a') # CREATES A NEW ONE!
puts fd.stat.ino
fd.flock(File::LOCK_EX) # DOESN'T BLOCK!

puts "Did it block #{Time.now}"

···

====================================

Here is the result...
ruby -w try.rb
6157824
Sleeping
6157824
Waiting for lock
Got lock
Open file Mon Aug 14 14:10:55 +1200 2006
6158309 <-------- Note new INODE NUMBER, Did it block Mon Aug 14 14:10:55 +1200 2006 <---- No it didn't!
__________________

John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : john.carter@tait.co.nz
New Zealand

Carter's Clarification of Murphy's Law.

"Things only ever go right so that they may go more spectacularly wrong later."

From this principle, all of life and physics may be deduced.

I think the unlink is your problem. In Unix when you unlink a file, it
disappears from the filesystem, but any processes that have an open handle
to the file will keep the file open until they close it. Any process that
tries to then open a file with the same name will end up creating a new one.
This happens in your example: process A creates and opens the lockfile, then
B opens it, then A unlinks the file and closes its handle to it. At this
point, the original file still exists (because B has it open and in fact
also has an advisory lock on it), but it has been unlinked from the
filesystem. So when C starts, it creates and locks a brand new file. The
first file doesn't disappear completely until B closes its handle to it.

Don't worry about leaving lock files around. If you really don't like it,
put it in /tmp or /dev/shm.

···

On 8/13/06, John Carter <john.carter@tait.co.nz> wrote:

So I'm using flock, but I have this noxious race condition when I try to
clean up the lock files.

I need a way of knowing if anything has a file open at the same time as
me, or I must always leave lock files lying around. Any suggestions?

Here is the sequence..
Process A: fd = open( "lockfile", 'a')
Process A: flock(fd, LOCK_EX)

Process A: do stuff in critical section...

Process B : fd = open( "lockfile", 'a')
Process B: flock(fd, LOCK_EX) // Blocks waiting for lock

Process A: Finishes, doesn't know about B, wants to clean up...
Process A: unlink fd
Process A: close(fd)

Process B: Unblocks, B grabs lock.

Process C : fd = open( "lockfile", 'a') // CREATES A NEW ONE!
Process C: flock(fd, LOCK_EX) // DOESN'T BLOCK!

John,

This might help:
http://homepage.mac.com/johnatl/.cv/johnatl/Sites/.Public/locker.rb-zip.zip

Regards,
   JJ

···

On Sun, 13 Aug 2006 22:16:13 -0400, John Carter <john.carter@tait.co.nz> wrote:

So I'm using flock, but I have this noxious race condition when I try to
clean up the lock files.

--
Using Opera's revolutionary e-mail client: Opera Web Browser | Faster, Safer, Smarter | Opera

Sigh! Just seems so...messy.

Sigh! I even thought of using whatever "fuser -v" uses, but strace tells
me it scans /proc! And that's gives me worse aesthetic collywobbles.

John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : john.carter@tait.co.nz
New Zealand

Carter's Clarification of Murphy's Law.

"Things only ever go right so that they may go more spectacularly wrong later."

From this principle, all of life and physics may be deduced.

···

On Mon, 14 Aug 2006, Francis Cianfrocca wrote:

Don't worry about leaving lock files around. If you really don't like it,
put it in /tmp or /dev/shm.

Sorry to jump in on the thread like this...
John are you using flock because it's the easiest & most available way to sync across processes? If you had easier access to one of the other sync primitives, would you be using it (i.e. a mutex)?
I'm just wondering what the dominant pattern is, and if we are using it because something better hasn't come along.

Peace,
Chris

···

On 14 Aug 2006, at 5:57 AM, John Carter wrote:

On Mon, 14 Aug 2006, Francis Cianfrocca wrote:

Don't worry about leaving lock files around. If you really don't like it,
put it in /tmp or /dev/shm.

Sigh! Just seems so...messy.

Sigh! I even thought of using whatever "fuser -v" uses, but strace tells
me it scans /proc! And that's gives me worse aesthetic collywobbles.

John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : john.carter@tait.co.nz
New Zealand

Carter's Clarification of Murphy's Law.

"Things only ever go right so that they may go more spectacularly wrong later."

From this principle, all of life and physics may be deduced.

You can check if the file you got a lock on is the one presently on the FS.
If the lockfile was unlinked, File.stat will fail; if another process created
a file with the same name, the ino will differ. You can detect both situations
and try again:

$ cat lock.rb

def lock(filename)
  dest = nil
  loop do
    dest.close if dest
    dest = File.open(filename, "ab")
    dest.flock(File::LOCK_EX)
    old_stat = dest.stat
    new_stat = File.stat(filename) rescue nil
    break if new_stat and
       old_stat.dev == new_stat.dev and
       old_stat.ino == new_stat.ino
  end
  yield
ensure
  File.unlink(filename) rescue nil
  dest.close rescue nil
end

LOCKFILE = "lockfile"

$stdout.sync = true

t = Time.new
fork do
  lock LOCKFILE do
    puts "A"
    p File.exist?(LOCKFILE)
    puts "Sleeping in A"
    sleep 10
  end
end

sleep 1
fork do
  lock LOCKFILE do
    puts "B"
    p File.exist?(LOCKFILE)
    puts "Sleeping in B"
    sleep 10
  end
end

sleep 15

puts "Attempting to get lock in C"
lock LOCKFILE do
  puts "C"
  p File.exist?(LOCKFILE)
end

puts "Total time: #{Time.new - t}"

$ ruby lock.rb
A
true
Sleeping in A
B
true
Sleeping in B
Attempting to get lock in C
C
true
Total time: 20.007932

···

On Mon, Aug 14, 2006 at 12:57:09PM +0900, John Carter wrote:

On Mon, 14 Aug 2006, Francis Cianfrocca wrote:

>Don't worry about leaving lock files around. If you really don't like it,
>put it in /tmp or /dev/shm.

Sigh! Just seems so...messy.

--
Mauricio Fernandez - http://eigenclass.org - singular Ruby

I'm have a generic Daemon class exactly like Process.fork but Daemon.spawn does things like...

* Check, via a lock file, that another copy of itself isn't running
   somewhere.

* Has a method to kill all copies of itself (via fuser)

* Detaches itself from the controlling terminal (making it self a true
   Daemon) so it doesn't die if parent process dies.

* Can wait until the dameon has started, can wait for it to die.

I suspect I could use fcntl and mandatory locking, but I'm not sure that
would help with this problem.

Ps: (What do Windowsy types do instead of "Process.fork"? )

John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : john.carter@tait.co.nz
New Zealand

Carter's Clarification of Murphy's Law.

"Things only ever go right so that they may go more spectacularly wrong later."

From this principle, all of life and physics may be deduced.

···

On Mon, 14 Aug 2006, Christopher Brown wrote:

John are you using flock because it's the easiest & most available way to sync across processes? If you had easier access to one of the other sync primitives, would you be using it (i.e. a mutex)?
I'm just wondering what the dominant pattern is, and if we are using it because something better hasn't come along.

Ps: (What do Windowsy types do instead of "Process.fork"? )

Cry, mostly.

:slight_smile:

Regards,

Bill

···

From: "John Carter" <john.carter@tait.co.nz>

Ah, gotcha. You really don't need strong cross-process synch, but just proof another of your processes already exists.

It sounds like you are following most of the guidance from APUE. In which case, on *nixes, why not write a pidfile in /var/run? It's a fairly standard way to do things.
Write the process id into /var/run/<application>.pid. Any launch of the daemon should check for the existence of this file first and for added hygiene, check the actual process id written into it against what's in the process table (or against /proc/<id> if on Linux). At the end of the day, it's much like the lockfile version, but gives a bit more information and some tools will work directly with the pidfile. On some Linux distros, there are tools that will create it for you and take the daemon managment out of your hands if you choose.

The Windows process model is quite a bit different, and really favors creating new threads (I guess not what you want here...) Fork / exec model isn't directly supported. I guess some toolkits attempt to fake it though. There's a nifty description of the cygwin implementation here: Highlights of Cygwin Functionality

I guess this is drifting off-topic from Ruby, so I'll shut up now.

Peace,
Chris

···

On 14 Aug 2006, at 6:49 AM, John Carter wrote:

On Mon, 14 Aug 2006, Christopher Brown wrote:

John are you using flock because it's the easiest & most available way to sync across processes? If you had easier access to one of the other sync primitives, would you be using it (i.e. a mutex)?
I'm just wondering what the dominant pattern is, and if we are using it because something better hasn't come along.

I'm have a generic Daemon class exactly like Process.fork but Daemon.spawn does things like...

* Check, via a lock file, that another copy of itself isn't running
  somewhere.

* Has a method to kill all copies of itself (via fuser)

* Detaches itself from the controlling terminal (making it self a true
  Daemon) so it doesn't die if parent process dies.

* Can wait until the dameon has started, can wait for it to die.

I suspect I could use fcntl and mandatory locking, but I'm not sure that
would help with this problem.

Ps: (What do Windowsy types do instead of "Process.fork"? )

John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : john.carter@tait.co.nz
New Zealand

Carter's Clarification of Murphy's Law.

"Things only ever go right so that they may go more spectacularly wrong later."

From this principle, all of life and physics may be deduced.

I have to say that in my opinion, none of the problems you're trying
to solve are particularly deep nor particularly new. Stick with the
tried and true rather than inventing new practice. On Unix, just use
flock in the standard way. Don't worry about lockfile clutter- that's
what /tmp and /var/run are for. Advisory locks vanish with file
descriptors which makes them graceful when things go wrong. Like any
"mutex" object, hold them for the shortest possible period of time and
make sure you're not susceptible to hanging while you hold one-
otherwise you will go mad once you are in production.

Don't use anything that involves scanning /proc: that locks you not
only to specific Unixes but even to specific versions. Don't use
mandatory locks: this isn't what they were designed for and they are
extremely painful when things go wrong.

On Windows, don't cry, just use CreateProcess. Windows has a strong
cross-process mutex, unlike Linux, so use it. It's heavyweight, but so
what? Nothing is heavier than a process, which is what you're trying
to sync in the first place.

···

On 8/14/06, John Carter <john.carter@tait.co.nz> wrote:

I'm have a generic Daemon class exactly like Process.fork but
Daemon.spawn does things like...

* Check, via a lock file, that another copy of itself isn't running
   somewhere.

* Has a method to kill all copies of itself (via fuser)

* Detaches itself from the controlling terminal (making it self a true
   Daemon) so it doesn't die if parent process dies.

* Can wait until the dameon has started, can wait for it to die.

I suspect I could use fcntl and mandatory locking, but I'm not sure that
would help with this problem.

Ps: (What do Windowsy types do instead of "Process.fork"? )

It depends on how close to fork() your needs are.

If your need is really closer to a fork & exec than just a fork, it's not too hard. I posted this code in another thread the other day, but here's a method I use in a test suite to allow the code to create new processes on Windows and *nix. It requires Daniel Berger's win32/process lib, but with that in hand -- no problem.

module IWATestSupport
   def self.create_process(args)
     @fork_ok = true unless @fork_ok == false
     pid = nil
     begin
       raise NotImplementedError unless @fork_ok
       unless pid = fork
         Dir.chdir args[:dir]
         exec(*args[:cmd])
       end
     rescue NotImplementedError
       @fork_ok = false
       begin
         require 'rubygems'
       rescue Exception
       end

       begin
         require 'win32/process'
       rescue LoadError
         raise "Please install win32-process to run all tests on a Win32 platform. 'gem install win32-process' or http://rubyforge.org/projects/win32utils&quot;
       end
       cwd = Dir.pwd
       Dir.chdir args[:dir]
       pid = Process.create(:app_name => args[:cmd].join(' '))
       Dir.chdir cwd
     end
     pid
   end
end

Kirk Haines

···

On Mon, 14 Aug 2006, Bill Kelly wrote:

From: "John Carter" <john.carter@tait.co.nz>

Ps: (What do Windowsy types do instead of "Process.fork"? )

Cry, mostly.

Hmm. One of the hallmarks of the Unix philosophy was/is the idea that
processes are 'cheap'.

I tend to think that people gravitate towards a threading solution a little
too reflexively these days and then get bitten by all sorts of concurrency
issues that can often be avoided via cooperating processes.

The Plan 9 papers offer some interesting insight into the process/thread
dichotomy.

Gary Wright

···

On Aug 14, 2006, at 6:44 AM, Francis Cianfrocca wrote:

It's heavyweight, but so what? Nothing is heavier than a process,
which is what you're trying to sync in the first place.

I have to say that in my opinion, none of the problems you're trying
to solve are particularly deep nor particularly new.

I know, that's why I assumed there was a standard solution I was missing.
Answer: Have a world writable standard clutter directory...and a
standard security hole where malware can do nasty things to temp
files and lock files.

On Windows, don't cry, just use CreateProcess. Windows has a strong
cross-process mutex, unlike Linux, so use it. It's heavyweight, but so
what? Nothing is heavier than a process, which is what you're trying
to sync in the first place.

ri CreateProcess
Nothing known about CreateProcess.

Where do I start looking for that?

John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : john.carter@tait.co.nz
New Zealand

Carter's Clarification of Murphy's Law.

"Things only ever go right so that they may go more spectacularly wrong later."

From this principle, all of life and physics may be deduced.

···

On Mon, 14 Aug 2006, Francis Cianfrocca wrote:

Hmm, you were responding to me so I thought I should clarify. One of
the questions upthread was about Windows, not Plan 9. On Unix just use
flock for process sync. On Windows there is a useable cross-process
mutex.

As far as threads are concerned, I bow to no one in my disdain for
them. (But I've been programming threaded apps since Win32 was in
beta, so I think I'm qualified to disdain them.)

···

On 8/14/06, gwtmp01@mac.com <gwtmp01@mac.com> wrote:

On Aug 14, 2006, at 6:44 AM, Francis Cianfrocca wrote:
> It's heavyweight, but so what? Nothing is heavier than a process,
> which is what you're trying to sync in the first place.

Hmm. One of the hallmarks of the Unix philosophy was/is the idea that
processes are 'cheap'.

I tend to think that people gravitate towards a threading solution a
little
too reflexively these days and then get bitten by all sorts of
concurrency
issues that can often be avoided via cooperating processes.

The Plan 9 papers offer some interesting insight into the process/thread
dichotomy.

Gary Wright

CreateProcess is a Windows API. So is CreateMutex, which you'll want
to look at. Daniel Berger's Win32 library may have Ruby wrappers for
these, so I'd have a look there too.

···

On 8/14/06, John Carter <john.carter@tait.co.nz> wrote:

Nothing known about CreateProcess.

Where do I start looking for that?

win32-process, found at http://rubyforge.org/projects/win32utils/ makes it easy to access CreateProcess.

Kirk Haines

···

On Tue, 15 Aug 2006, John Carter wrote:

ri CreateProcess
Nothing known about CreateProcess.

Where do I start looking for that?

I guess it wasn't clear to me that you were responding to
the situation with respect to windows. It seemed like a more general
comment.

Lots of people seem to think Unix threads are heavyweight also.

Gary Wright

···

On Aug 14, 2006, at 12:45 PM, Francis Cianfrocca wrote:

Hmm, you were responding to me so I thought I should clarify. One of
the questions upthread was about Windows, not Plan 9. On Unix just use
flock for process sync. On Windows there is a useable cross-process
mutex.

Depends on the Unix. Solaris threads (not LWPs) are light as a feather.
Linux 2.6 threads are almost as heavy as processes. Linux 2.4 threads are
almost unusable.

···

On 8/14/06, gwtmp01@mac.com <gwtmp01@mac.com> wrote:

On Aug 14, 2006, at 12:45 PM, Francis Cianfrocca wrote:

> Hmm, you were responding to me so I thought I should clarify. One of
> the questions upthread was about Windows, not Plan 9. On Unix just use
> flock for process sync. On Windows there is a useable cross-process
> mutex.
>

I guess it wasn't clear to me that you were responding to
the situation with respect to windows. It seemed like a more general
comment.

Lots of people seem to think Unix threads are heavyweight also.

Gary Wright

Francis Cianfrocca wrote:

Depends on the Unix. Solaris threads (not LWPs) are light as a feather.
Linux 2.6 threads are almost as heavy as processes. Linux 2.4 threads are
almost unusable.

Almost unusable? How are they worse than "almost as heavy as processes?"

:slight_smile:

The scheduler was not nearly as scalable in 2.4 as it is now, based on my
experiences. Try to spin five hundred threads on a 2.4 kernel and you won't
be doing much besides thrashing them around. Even Windows can do more than
that (a lot more, actually).

···

On 8/15/06, M. Edward (Ed) Borasky <znmeb@cesmail.net> wrote:

Francis Cianfrocca wrote:
> Depends on the Unix. Solaris threads (not LWPs) are light as a feather.
> Linux 2.6 threads are almost as heavy as processes. Linux 2.4 threads
are
> almost unusable.
Almost unusable? How are they worse than "almost as heavy as processes?"

:slight_smile: