Geek Foibles

AppleVNCServer crashing when connected to
October 5, 2009, 8:17 pm
Filed under: Uncategorized | Tags: ,

For the past few weeks, AppleVNCServer (aka screen sharing) crashes when you connect to it.  It’s on a Mac mini running Mac OS X Server 10.5.8.  Here’s the relevant bit from system.log:

Oct  3 14:17:35 minnie ReportCrash[12942]: Formulating crash report for process AppleVNCServer[12874]
Oct  3 14:17:36 minnie[264] ([12874]): Exited abnormally: Segmentation fault
Oct  3 14:17:37 minnie ReportCrash[12942]: Saved crashreport to /Users/will/Library/Logs/CrashReporter/AppleVNCServer_2009-10-03-141730_minnie.crash using uid: 501 gid: 20, euid: 501 egid: 20

Nothing too interesting there, so here’s the crash log it mentions:

Process:         AppleVNCServer [12874]
Path:            /System/Library/CoreServices/RemoteManagement/AppleVNCServer.bundle/Contents/MacOS/AppleVNCServer
Identifier:      AppleVNCServer
Version:         ??? (???)
Code Type:       X86 (Native)
Parent Process:  launchd [264]

Date/Time:       2009-10-03 14:17:30.662 -0400
OS Version:      Mac OS X Server 10.5.8 (9L34)
Report Version:  6
Anonymous UUID:  1DFDDD4A-5259-4417-A19F-8DA81EA911FC

Exception Codes: 0x000000000000000d, 0x0000000000000000
Crashed Thread:  4

Thread 0:
0   libSystem.B.dylib             	0x90d51286 mach_msg_trap + 10
1   libSystem.B.dylib             	0x90d58a7c mach_msg + 72
2      	0x9098de7e CFRunLoopRunSpecific + 1790
3      	0x9098eb04 CFRunLoopRun + 84
4      	0x00005a4f 0x1000 + 19023
5      	0x00002675 0x1000 + 5749
6      	0x00025636 0x1000 + 149046
7      	0x00001fd9 0x1000 + 4057

Thread 1:
0   libSystem.B.dylib             	0x90d819c6 kevent + 10
1   ...ple.CoreServices.CarbonCore	0x95789057 PrivateMPEntryPoint + 56
2   libSystem.B.dylib             	0x90d82155 _pthread_start + 321
3   libSystem.B.dylib             	0x90d82012 thread_start + 34

Thread 2:
0   libSystem.B.dylib             	0x90d883ca select$DARWIN_EXTSN$NOCANCEL + 10
1      	0x00004c9b 0x1000 + 15515
2   ...ple.CoreServices.CarbonCore	0x95789057 PrivateMPEntryPoint + 56
3   libSystem.B.dylib             	0x90d82155 _pthread_start + 321
4   libSystem.B.dylib             	0x90d82012 thread_start + 34

Thread 3:
0   libSystem.B.dylib             	0x90d512da semaphore_timedwait_trap + 10
1   ...ple.CoreServices.CarbonCore	0x9579c47b MPWaitOnSemaphore + 125
2      	0x00006be5 0x1000 + 23525
3   ...ple.CoreServices.CarbonCore	0x95789057 PrivateMPEntryPoint + 56
4   libSystem.B.dylib             	0x90d82155 _pthread_start + 321
5   libSystem.B.dylib             	0x90d82012 thread_start + 34

Thread 4 Crashed:
0      	0x00037a1e 0x1000 + 223774
1      	0x000157c8 0x1000 + 83912
2      	0x00014a5f 0x1000 + 80479
3      	0x00008fa4 0x1000 + 32676
4   ...ple.CoreServices.CarbonCore	0x95789057 PrivateMPEntryPoint + 56
5   libSystem.B.dylib             	0x90d82155 _pthread_start + 321
6   libSystem.B.dylib             	0x90d82012 thread_start + 34

Thread 5:
0   libSystem.B.dylib             	0x90d51286 mach_msg_trap + 10
1   libSystem.B.dylib             	0x90d58a7c mach_msg + 72
2     	0x911a94b7 io_connect_method + 592
3     	0x911333c6 IOConnectCallMethod + 300
4	0x00782c95 glrReadPixels + 1973
5   GLEngine                      	0x005ff144 glReadPixels_Exec + 2004
6   libGL.dylib                   	0x93eecaa9 glReadPixels + 73
7      	0x000155ee 0x1000 + 83438
8      	0x00009d87 0x1000 + 36231
9   ...ple.CoreServices.CarbonCore	0x95789057 PrivateMPEntryPoint + 56
10  libSystem.B.dylib             	0x90d82155 _pthread_start + 321
11  libSystem.B.dylib             	0x90d82012 thread_start + 34

Thread 6:
0   libSystem.B.dylib             	0x90d5846e __semwait_signal + 10
1   libSystem.B.dylib             	0x90d82dcd pthread_cond_wait$UNIX2003 + 73
2   libGLProgrammability.dylib    	0x93f43b32 glvmDoWork + 162
3   libSystem.B.dylib             	0x90d82155 _pthread_start + 321
4   libSystem.B.dylib             	0x90d82012 thread_start + 34

Thread 4 crashed with X86 Thread State (32-bit):
  eax: 0x00807c00  ebx: 0x00000000  ecx: 0x000b5000  edx: 0xffffeaa8
  edi: 0x000b5000  esi: 0x125d40a8  ebp: 0xb0226c28  esp: 0xb0226854
   ss: 0x0000001f  efl: 0x00010202  eip: 0x00037a1e   cs: 0x00000017
   ds: 0x0000001f   es: 0x0000001f   fs: 0x0000001f   gs: 0x00000037
  cr2: 0x00037a10

Binary Images:
    0x1000 -    0x4dff7 3.3.1 (3.3.1) <5b4a7fdb0c13de88d2d0bd7c309047ed> /System/Library/CoreServices/RemoteManagement/AppleVNCServer.bundle/Contents/MacOS/AppleVNCServer
   0xcb000 -    0xe7ff7  GLRendererFloat ??? (???) <7badea5e2b8167c0e6391623bb46140a> /System/Library/Frameworks/OpenGL.framework/Versions/A/Resources/GLRendererFloat.bundle/GLRendererFloat
  0x5b2000 -   0x737fe3  GLEngine ??? (???) <052e02d9a452a45d014ffbd2a84a4e7c> /System/Library/Frameworks/OpenGL.framework/Resources/GLEngine.bundle/GLEngine
  0x765000 -   0x7beff7 1.5.48 (5.4.8) <806d3030842b46995c46c9059046f2fc> /System/Library/Extensions/AppleIntelGMA950GLDriver.bundle/Contents/MacOS/AppleIntelGMA950GLDriver
0x8fe00000 - 0x8fe2db43  dyld 97.1 (???) <458eed38a009e5658a79579e7bc26603> /usr/lib/dyld
0x9020b000 - 0x90264ff7  libGLU.dylib ??? (???) <64d010e31d7596bd8f9edc6e027d1d0c> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
0x9034d000 - 0x90354ffe  libbsm.dylib ??? (???)  /usr/lib/libbsm.dylib
0x9036d000 - 0x9077dfef  libBLAS.dylib ??? (???) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
0x9077e000 - 0x907dbffb  libstdc++.6.dylib ??? (???) <04b812dcec670daa8b7d2852ab14be60> /usr/lib/libstdc++.6.dylib
0x907dc000 - 0x907dcffd 1.4.2 (Accelerate 1.4.2) /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
0x907dd000 - 0x907ebffd  libz.1.dylib ??? (???) <5ddd8539ae2ebfd8e7cc1c57525385c7> /usr/lib/libz.1.dylib
0x90890000 - 0x9091afe3 1.4.8 (1.4.8)  /System/Library/PrivateFrameworks/DesktopServicesPriv.framework/Versions/A/DesktopServicesPriv
0x9091b000 - 0x90a4efe7 6.5.7 (476.19)  /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
0x90a94000 - 0x90a94ff8 34 (34) <8f910fa65f01d401ad8d04cc933cf887> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
0x90cfe000 - 0x90d4fff7 1.7.1 (???)  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
0x90d50000 - 0x90eb7ff3  libSystem.B.dylib ??? (???)  /usr/lib/libSystem.B.dylib
0x90eb8000 - 0x90f4bfff 101.3 (86)  /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/Ink.framework/Versions/A/Ink
0x90f52000 - 0x90f8cfe7 1.2 (62) /System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/CoreUI
0x90f8d000 - 0x90f96fff 3.7.24 (3.7.24)  /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/SpeechRecognition.framework/Versions/A/SpeechRecognition
0x91073000 - 0x91074ffc  libffi.dylib ??? (???)  /usr/lib/libffi.dylib
0x910f3000 - 0x910f8fff 1.2.4 (85)  /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/CommonPanels.framework/Versions/A/CommonPanels
0x9112e000 - 0x911bbff7 1.5.2 (???) <7a3cc24f78f93931731203854ae0d891> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
0x911bc000 - 0x91243ff7  libsqlite3.0.dylib ??? (???)  /usr/lib/libsqlite3.0.dylib
0x91244000 - 0x912d1ff7 292 (292)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
0x912d2000 - 0x912f0ff3 3.5.6 (3.5.6) <48d78074835db0d5fe5b632991c21b1b> /System/Library/Frameworks/DirectoryService.framework/Versions/A/DirectoryService
0x912f1000 - 0x91310ffa  libJPEG.dylib ??? (???) <50b881dd5a5795d38405c9c88c2806fa> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
0x91311000 - 0x91350fef  libTIFF.dylib ??? (???) <801873cbd85ba7bdfe7646fe97a54ca3> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
0x92330000 - 0x92501ffb 5.0.5 (36371) <1f7f48b36bc90d114220cc81e4e4694f> /System/Library/Frameworks/Security.framework/Versions/A/Security
0x92666000 - 0x92693feb  libvDSP.dylib ??? (???)  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
0x92694000 - 0x9273bfec 438.14 (438.14) <5f9ee0430b5f6319f18d9b23e777e0d2> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
0x9278e000 - 0x9278effd 3.4.2 (vecLib 3.4.2) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
0x9278f000 - 0x927a4ffb 5.0.2 (5.0.2) /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/ImageCapture.framework/Versions/A/ImageCapture
0x92978000 - 0x92978ffb 1.0 (8) /System/Library/PrivateFrameworks/InstallServer.framework/Versions/A/InstallServer
0x92cc7000 - 0x92cc7ffa 32 (32) <2fcc8f3bd5bbfc000b476cad8e6a3dd2> /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
0x92cc8000 - 0x92fd0fe7 1.5.6 (???)  /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0x92fd1000 - 0x92fedff3  libPng.dylib ??? (???) <271373dd41f56369a3dfca0ed2be579a> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
0x92fee000 - 0x93017fff  libcups.2.dylib ??? (???) <3f0976287f63781607864f6586ad7229> /usr/lib/libcups.2.dylib
0x93018000 - 0x9303cfff  libxslt.1.dylib ??? (???)  /usr/lib/libxslt.1.dylib
0x9303d000 - 0x9303dffc 1.5 (1.5) /System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
0x93075000 - 0x930f4ff5 1.2.2 (1.2.2) <3b5f3ab6a363a4d8a2bbbf74213ab0e5> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
0x931ef000 - 0x93296feb 3.11.56 (???)  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/QD.framework/Versions/A/QD
0x932c1000 - 0x93413ff3 1.5.2 (1.5.2) /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
0x93414000 - 0x9342afff 1.0.0 (1.0.0)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
0x93453000 - 0x9346bff7 1.6.0 (20.0)
0x934a7000 - 0x93587fff  libobjc.A.dylib ??? (???) <3ca288b625a47bbcfe378158e4dc328f> /usr/lib/libobjc.A.dylib
0x93590000 - 0x93594fff  libmathCommon.A.dylib ??? (???) /usr/lib/system/libmathCommon.A.dylib
0x93596000 - 0x93613fef  libvMisc.dylib ??? (???) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
0x93614000 - 0x93652fff  libGLImage.dylib ??? (???) <2e570958595e0c9c3a289158223b39ee> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
0x93653000 - 0x93a11fea  libLAPACK.dylib ??? (???) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
0x93a12000 - 0x93a18fff 218.0.3 (220.2) <8c541d587e4068a5fe5a5ce8ee208516> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/Print.framework/Versions/A/Print
0x93a19000 - 0x93a1bfff 3.0 (30817) <2b2854123fed609d1820d2779e2e0963> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/SecurityHI.framework/Versions/A/SecurityHI
0x93a1c000 - 0x93a2cffc 1.6.5 (1.6.5)  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/LangAnalysis.framework/Versions/A/LangAnalysis
0x93a2d000 - 0x93ca9fe7 6.5.9 (677.26)  /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
0x93caa000 - 0x93d24ff8 5.5.4 (245.6) <9ae833544b8249984c07544dbe6a97fa> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/PrintCore.framework/Versions/A/PrintCore
0x93d25000 - 0x93ddffe3 228 (228)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
0x93ee7000 - 0x93ee7fff 136 (136) <98a5e3bc0c4fa44bbb09713bb88707fe> /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
0x93ee8000 - 0x93eeaff5  libRadiance.dylib ??? (???)  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
0x93eeb000 - 0x93ef7ffe  libGL.dylib ??? (???) /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
0x93ef8000 - 0x93f10fff 1.2.8 (???)  /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/OpenScripting.framework/Versions/A/OpenScripting
0x93f11000 - 0x93f1bfeb 3.9.2 (3.9.2) <0f2ba6e891d3761212cf5a5e6134d683> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/CarbonSound.framework/Versions/A/CarbonSound
0x93f1c000 - 0x943edfbe  libGLProgrammability.dylib ??? (???)  /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLProgrammability.dylib
0x943ee000 - 0x94430fef 3.5.2 (163) <7f4f1766414a511bf5bc68920ac85a88> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/NavigationServices.framework/Versions/A/NavigationServices
0x94456000 - 0x94485fe3 402.3 (402.3)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
0x94486000 - 0x94503feb 3.1.2 (3.1.2) <782a08c44be4698597f4bbd79cac21c6> /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
0x94504000 - 0x9450bfe9  libgcc_s.1.dylib ??? (???)  /usr/lib/libgcc_s.1.dylib
0x9450c000 - 0x9452afff  libresolv.9.dylib ??? (???) <0e26b308654f33fc94a0c010a50751f9> /usr/lib/libresolv.9.dylib
0x94592000 - 0x945a2fff 3.7.1 (3.7.1) <9a71429c74ed6ca43eb35e1f78471b2e> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
0x945a3000 - 0x94636ff3 3.7 (???)  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
0x94637000 - 0x9463ffff 2.2.1 (2.2.1)  /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
0x9465b000 - 0x946a4fef 10.5.8 (398.26)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
0x946a5000 - 0x946a5ff8 6.5 (???)  /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
0x946a6000 - 0x946aafff  libGIF.dylib ??? (???) <3c7100e80b7f7ca8809cf9512c1a6004> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
0x946ab000 - 0x9475dffb  libcrypto.0.9.7.dylib ??? (???) <8ac6abef4b3bb125c8bf84634421bcee> /usr/lib/libcrypto.0.9.7.dylib
0x94897000 - 0x94978ff7  libxml2.2.dylib ??? (???) <4d54971007e14f8d80e6889ee1111f21> /usr/lib/libxml2.2.dylib
0x95206000 - 0x95206ffd 3.4.2 (vecLib 3.4.2) /System/Library/Frameworks/vecLib.framework/Versions/A/vecLib
0x95207000 - 0x95340ff7  libicucore.A.dylib ??? (???)
0x95341000 - 0x956defef 1.5.8 (1.5.8) <18113e06d296230d63a63b58baf35f55> /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
0x95708000 - 0x9573ffff 1.9.2 (1.9.2) <41d5aeffefc6d19d471f51ae0b15024f> /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
0x95740000 - 0x95a1aff3 786.11 (786.14)  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0x95a1b000 - 0x95a1efff 1.1 (36)  /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/Help.framework/Versions/A/Help
0x95aa7000 - 0x95b6eff2 3.0 (3.0) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
0x95b6f000 - 0x9636dfef 6.5.9 (949.54) <4df5d2e2271175452103f789b4f4d8a8> /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
0x967de000 - 0x968c6ff3 100.2 (186.2) <44df326fea0236718f5ed64084e82270> /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
0x96936000 - 0x96961fe7  libauto.dylib ??? (???) <4f3e58cb81da07a1662c1f647ce30225> /usr/lib/libauto.dylib
0x9699b000 - 0x96a4bfff 6.0.13 (6.0.13)  /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
0x96a4c000 - 0x96aa8ff7 68 (1.1.3)  /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HTMLRendering.framework/Versions/A/HTMLRendering
0x96cae000 - 0x96cbbfe7 1.5.10 (1.5.10)  /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
0x96cbf000 - 0x96e07ff7 2.0.6 (2.0.6) <7f73ef328c8e8566f3f204b5a540a7f0> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/ImageIO
0x96e08000 - 0x96e62ff7 2.0.4 (???)  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreText.framework/Versions/A/CoreText
0x96e79000 - 0x96f44fef 4.5.3 (4.5.3) /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSync.framework/Versions/A/ColorSync
0x96f45000 - 0x975e5fef 1.409.4 (???)  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
0xfffe8000 - 0xfffebfff  libobjc.A.dylib ??? (???) /usr/lib/libobjc.A.dylib
0xffff0000 - 0xffff1780  libSystem.B.dylib ??? (???) /usr/lib/libSystem.B.dylib

Here’s where I reach the limits of my knowledge.  It looks like the system is pointing its finger at thread 4 as the culprit, and thread 4 looks like it’s loaded up the libSystem and Carbon libraries.  So perhaps one of those core system libraries are corrupted?  I tried re-applying the 10.5.8 combo update (since that will overwrite a lot of libraries with the latest version, just in case) to no effect, and I removed all the VNC- and Remote Desktop-related files in /Library/Preferences/ without any luck, either.

Obviously I could do an archive or erase and install and it’d take care of the issue, but I’m interested in actually sussing out the cause and pinpointing it.  Any suggestions are welcome at this point!

UPDATED – Nov. 19th, 2009
A few days after posting this I noticed that while connecting to the VNC server with Mac OS X’s built-in VNC client (called by choosing “Connect to Server…” from the “Go” menu in Finder and providing it a “vnc://” URL) incurs the crash of the server, connecting to it with Apple’s Remote Desktop application doesn’t.  So, I’ve got a workaround, but still no solution.  I’m going to try to chase this down in the next couple weeks, hopefully I’ll be able to pinpoint the cause.

Making your Hurricane Electric IPv6 tunnel start automatically on FreeBSD
August 7, 2009, 8:45 pm
Filed under: Uncategorized | Tags: ,

In a recent fit of masochism, I decided to get all of my systems onto IPv6.  I get the impression that the worldwide IPv6 transition is going to continue to be a very slow and tedious one, but by God I don’t intend to be on the trailing edge of it, so in I dove.  Hurricane Electric has a quite good setup featuring a nice, technical site that caters to us geeks without falling into the common trap of being difficult to use.  I’ll assume that you’ve already created your tunnel on their site.  It should be a “regular” tunnel and you should provide them with your public IPv4 address, even if the machine you’ll be using it with is NAT-ed.

It’s worth mentioning here that if you’re behind NAT, it is my impression that your router needs to be able to pass “protocol 41” traffic in order for this to work.  I use pfSense for my router, and it has a special field where I can provide the LAN IP of the machine incoming protocol 41 traffic should be forwarded to.  Other router software may have similar abilities, but yet others may be unable to do this.  There are apparently other ways of getting IPv6 tunnels through routers, such as AYIYA or Cisco’s GRE, but I won’t be getting into them here.

So, back to your tunnel.  At the bottom of the “tunnel details” page there’s a not totally obvious but very handy menu that you can use to get instructions on how to set up your tunnel under several popular operating systems:

Example OS Configurations

The only problem is, these instructions only tell you how to fire up the tunnel once.  It doesn’t tell you how to have your system start the tunnel automatically during start up.  So, after looking at a few search results and then a bit of tinkering, I’m happy to say I figured out how to have FreeBSD start your tunnel during startup.  Here’s what I ended up putting into my /etc/rc.conf file:

ipv6_defaultrouter="<server IPv6 address>"
gifconfig_gif0="<client IPv4 address> <server IPv4 address>"
ipv6_ifconfig_gif0="<client IPv6 address> <server IPv6 address> prefixlen 128"

Now, note that <client IPv4 address> should be your local system’s assigned IPv4 address, as in the one that shows up when you run ifconfig.  It’s okay if this is a private, NAT-ed IP address.  In fact, if you are using NAT, using your network’s public-facing IP address will not work.  Use the IP address your system is actually assigned, regardless of whether it’s a public or private IP.

In addition to the IPv6 address Hurricane Electric assigns you for your side of the tunnel (listed on their site as the “Client IPv4 address”), they kindly give you a whole /64 subnet to have your way with.  If you want to assign any of those addresses to your system in addition to the one we already do above, add a line like this to /etc/rc.conf for the first address:

ipv6_ifconfig_gif0_alias0="<first subnet address> prefixlen 64"

Your subnet address is listed as “Routed /64” on your “tunnel details” page.  If, for example, it’s listed as 2001:618:400:33bb::/64, your first address in that subnet could be: 2001:618:400:33bb::1 For additional addresses, you’ll need to increment the alias number, e.g. alias1, alias2, etc.

So that’s it!  Now your FreeBSD box should be tunneling IPv6 traffic to and from Hurricane Electric’s network.  You can test it out like so:


Overcoming’s lack of verbosity
April 21, 2009, 4:35 am
Filed under: Uncategorized

I’m generally a fan of, the simple e-mail client that is a part of Mac OS X.  I find that simplicity in software is a virtue, as it usually results in small, reliable programs. is, for the most part, a good example of that.  It is easy to use, fast and is less prone to data corruption than alternatives like Entourage.

Sometimes its simplicity can be a liability, though.  This is especially true when trying to troubleshoot connection issues, as will often not relay error messages from the server it’s communicating with.  For example, if you try sending an attachment that’s too big, the outgoing mail server that you try to send it through will tell, “Hey, this is too big,” but will only tell you that it couldn’t send the message.  It’s left up to you to try to figure out why.

In my case, a client was trying to forward an e-mail and its attachments that he’d received in from his Gmail account.  I could watch the forward’s send progress in’s activity window, but shortly after it got to 96%, would simply report that it can’t send the message without any elaboration. SMTP error

Historically I would fire up tcpdump in Terminal in order to observe (“sniff”) the conversation between and the server, thus seeing any error messages that pass between the two.  Servers are usually very explicit in their messages, so usually when you finally see the message you know right away what’s going wrong and how to fix it.  Trick is, tcpdump is only useful if you’re not using SSL or TLS to encrypt the connection to the server.  If you are, everything will be garbled and unintelligible.  Often I could simply turn off encryption temporarily and sniff the connection long enough to glean something useful, but services like Gmail will not work at all unless encryption is enabled.

After a bit of googling, I finally came across a post on Erik’s Lab that explained how to sniff any communications, regardless of encryption.  Apparently there is a hook built into that repeats a specified port’s traffic to the console.  So, if you launch from Terminal using the following command (in our case sniffing port 25 to troubleshoot outgoing mail), you will see all the relevant data:

/Applications/ -LogActivityOnPort 25

Leave Terminal open, since that’s where you’ll see the data appear.  I went ahead and sent the message again, and lo, after watching gobs of attachment data fly past, the following message appeared:

552-5.7.0 Our system detected an illegal attachment on your message. Please
552-5.7.0 visit to
552 5.7.0 review our attachment guidelines.

I removed the attached ZIP file and then the e-mail sent without a problem.

Front Row does not work on Mac OS X Server
March 13, 2009, 3:05 am
Filed under: Uncategorized | Tags: , , ,

Really, to be more accurate, I should say that on Mac OS X Server, Front Row has not been updated and therefore does not understand libraries from the last few versions of iTunes. The reason for this is simple.  Front Row is really just a fancy, minimal face for several different data sources, most significantly iTunes.  iTunes keeps track of all its media via the file named “iTunes Library” found in your “iTunes” folder.  Each new version of iTunes gets all sorts of new bells and whistles, thus requiring the library file to keep track of new kinds of information.  As a result, the library file’s format changes quite frequently.

Now, in order for Front Row to display media in the iTunes library, it has to understand the library file’s format.  So, it has to be updated every time the iTunes library format is.  This is why there are often new versions of Front Row released alongside new versions of iTunes.  If the new version of iTunes uses a new library format, Front Row has to be tweaked to work with it.

This is all fine and dandy, except for the inexplicable fact that Apple hasn’t released any updates to Front Row for Mac OS X Server in quite some time.  In fact, I’m unable to determine if they’ve ever released any for Leopard Server.  Mac OS X gets them literally the moment they’re needed, released simultaneously with the new versions of iTunes that necessitate it.  Those same iTunes updates are available for Server, but for some reason the corresponding Front Row updates don’t materialize.  You can download the one for Mac OS X, but Installer won’t install it because it’s not intended for Server.

In my own setup, my iTunes library lives on a drive connected to a Mac mini running Mac OS X Server.  I do all my work on my MacBook Pro, including my iTunes playing/ripping/purchasing/etc., and it just accesses the library via the network.  I’m running the most recent version of iTunes most of the time.  The mini is connected to my HDTV so that I can use Boxee and Front Row, but when Server’s copy of Front Row is too old to understand the newer library format the latest iTunes uses, Front Row displays an empty library and the whole point is lost.

The solution is simply to download the update for Mac OS X, open the package with Pacifist and install it anyway per the instructions over on the MacRumors Forums.  I haven’t used it a lot since forcing the update this way, but so far things seem to work perfectly.

I honestly can’t imagine why Apple is neglecting Front Row on Mac OS X Server, since it doesn’t seem like it’s really any more effort than what they’re already doing.  I can’t imagine Front Row for Mac OS X is especially different than Front Row for Server, if at all.  Obviously Front Row probably doesn’t see a lot of use on Mac OS X Server, but it is a standard part of Server’s install so it seems odd to intentionally neglect it.  Chalk it up as another one of those things Apple does that makes sense to no one but them.

amavisd not starting after restarting Mac OS X Server
February 6, 2009, 4:31 am
Filed under: Uncategorized | Tags: , , , ,

I had a recent spot of trouble with Mac OS X Server for a client who has mail services running.  Due to a hardware issue I had to force off the server, but when I started it back up I noticed that while Postfix would accept mail from senders, it wouldn’t actually deliver it to the destination mailbox.  Instead, I’d get this message in mail.log:

postfix/error[7063]: 56118A83885: to=<>, relay=none, delay=4776, delays=4776/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to[]: Connection refused)

The complaint that it wasn’t delivering mail due to an inability to connect to the local host suggested some sort of mail filter not running, of which there are two on Mac OS X Server: clamav (used for virus scanning) and amavisd (used for spam filtering).  In my case it was easy to narrow it down, since clamav wasn’t enabled.  I checked amavis.log and found that it hadn’t been updated since I’d restarted, so it seemed clear it wasn’t launching.  By looking in system.log I could see launchd was trying to start it repeatedly since it was enabled in Server Admin, but it kept failing:

Feb  5 23:59:36 hq org.amavis.amavisd[5404]: Can't send SIG 0 to process [68]: Operation not permitted at /usr/bin/amavisd line 11203.
Feb  5 23:59:36 hq[1] (org.amavis.amavisd[5404]): Exited with exit code: 1
Feb  5 23:59:36 hq[1] (org.amavis.amavisd): Throttling respawn: Will start in 10 seconds

It was trying to kill process 68, which in my case was kadmind.  This didn’t make sense since kadmind deals with Kerberos, something spam filtering has nothing to do with.  Fortunately amavisd is a Perl script and it was giving me the line number, so I was able to open it and hunt down the offending code.  I found that when amavisd starts, it looks to see if there’s already a PID file (located at /var/amavis/, gets the PID out of that file and then tries to kill said PID.  So, if another copy of amavisd is already running when you start it, it kills the old one so that they don’t conflict.  Problem is, in my case the PID file was stale: it hadn’t been cleaned up after amavisd’s last run since I’d had to force off the system, and now the PID was in use by another, unrelated process that it doesn’t have permission to kill.

amavisd does check to see if the PID file was created before the last system startup, which would allow it to ignore stale PID files, but it only does this after trying to kill the old process.  This bug was fixed in December in version 2.6.2 of amavis, but unfortunately for Mac OS X Server users, Apple uses version 2.5.1 from May of 2007.

Removing the stale PID file resolved the issue.  Its removal allowed launchd’s next automatic attempt to start amavisd to succeed, then a few minutes later Postfix’s mail queue manager started being able to deliver all the mail queued during the downtime to the appropriate mailboxes.

Front Row and iTunes on two systems sharing one library
November 26, 2008, 2:31 am
Filed under: Uncategorized | Tags: , , ,

One of the keys to my home media center project is the iTunes library.  All my music and all my TV shows are in iTunes, so all facets of the media center orbit around it.  Trick is, my media center and my personal computer are two different machines.

Fortunately, it’s pretty easy to share a single iTunes library between two (or more) computers.  While iTunes used to insist on your iTunes library living in ~/Music/iTunes, as of version 7 you can  keep it wherever you like and simply tell iTunes where it is.  So in my case I keep my iTunes library on my Mac Mini media center, since it does double duty as my file server.  On my personal computer I mounted the share it’s on and directed iTunes to it, and presto, it functions exactly as if it were residing locally.  (Having a gigabit Ethernet link between the two certainly helps keep things snappy, though even when I switch seamlessly to WiFi iTunes can still play TV shows smoothly.)  While I don’t run iTunes on the Mini, I do use Front Row, which relies on iTunes’ preferences for the location of the iTunes library.  So, I fired up iTunes on the Mini once just to tell it where the library was so that Front Row can use it.  Now Front Row on the Mini shares the same library with iTunes on my MacBook Pro.  Front Row doesn’t even seem to mind if iTunes is running simultaneously on the other machine, I assume because Front Row only needs read-only access.

There are two small quirks to this setup.  First, Front Row doesn’t update play counts on the shared library.  The reasoning for this is unclear to me.  Perhaps there’s a unique system ID in the library file that identifies the computer that’s last used it, and if it conflicts with the ID of the system Front Row is running on, it doesn’t modify it.  I’m going to delve into that further.  Second, Front Row reads the library only when it first launches, so when I download a new Daily Show episode on my MacBook Pro, Front Row on the Mini doesn’t know about it.  One would think that exiting out of Front Row back to the desktop, then going back in again would resolve this, but after some frustration I discovered that Front Row doesn’t actually quit when you go back to the desktop.  It stays running in the background, and thus the library is still cached in it.  To get Front Row to rescan the iTunes library, you need to kill it in Terminal or Activity Monitor.  The following command in Terminal will suffice:

killall "Front Row"

Once you’ve done that, Front Row can be relaunched and it will reflect the most recent changes to the library.

Why I love NFS
September 8, 2008, 3:20 am
Filed under: Uncategorized | Tags: , , , ,

There are three major protocols in the filesystem sharing world.  The most common is of course SMB, that which is used between most Windows systems.  There is AFP, Apple’s equivalent, and finally there is my personal favorite, NFS.  NFS is used primarily on UNIX-like systems, and fortunately Mac OS X falls into that category.  For Mac users, AFP is usually the best choice just because of how well-integrated into the OS the protocol is.  However, the demise of resource forks that corresponded with the advent of Mac OS X has made the advantages of AFP on the Mac platform a bit more muted.

NFS lacks some important things that AFP and SMB both provide.  Security is probably the most immediately concerning issue, as there is no user authentication.  While AFP and SMB both establish a TCP connection that first requires interactive authentication, NFS instead uses UDP, which as a stateless protocol doesn’t even have connections.  Packets just get passed back and forth between the hosts, there’s no “connected” state between them.  (Version 4 of the NFS standard does add TCP support, but that doesn’t help my soon-to-be-mentioned problem.)  Authentication is host-based, meaning you tell the server what hosts can access what shares and with what privileges.

NFS’s use of stateless UDP can simultaneously be a virtue.  In my home, all my work gets done on a MacBook Pro.  I like to move my computer around, but also need high-speed access to large amounts of data.  I keep a sizable iTunes library on a file server, along with my Parallels virtual machines and a variety of other large pieces of data.  So, I have an 802.11n wireless network in addition to gigabit Ethernet.  I keep the gigabit plugged in when I’m at my desk, then switch to wireless when I want to move elsewhere.  However, this solution of dual interfaces introduces a new problem: when I switch interfaces, it kills most of my connections.  This means that, if I’m using an AFP or SMB file server, I have to disconnect from it before I physically disconnect.  This means quitting iTunes, stopping any video encodes I have going in the background, stopping all my virtual machines, etc.  It’s not the end of the world, but it definitely disrupts my workflow.  It would be really nice to avoid this hassle, and fortunately NFS provides a way to do just that.

To explain how, it helps to understand a few basic things about how traffic moves via TCP and UDP.  When you establish a TCP connection, it is between two IP addresses.  Each network interface has its own IP address, so TCP connections are bound to particular interfaces.  For example, if I’m connected to both Ethernet and wireless, let’s say my Ethernet interface has the IP and my wireless interface has  I of course have my interface priority set to prefer Ethernet over wireless, so if I start downloading something in my web browser (which uses TCP), it will start doing so over the Ethernet.  Let’s pretend the server I’m downloading from is  So, the TCP connection is established between and  Now, if part-way through this download I disconnect my Ethernet cable, the connection will be lost and the download will abort.  The fact that I also have a simultaneous wireless connection doesn’t matter, since that interface has a different IP address and the connection does not involve that address.  I could, however, attempt the download again immediately after its failure and it would work, this time with a connection established via the wireless interface.

Now, let’s start the example over, but instead of downloading a file via the web, let’s transfer a file via NFS.  First, in order for this to work, the NFS server has to be configured to allow access from both the Ethernet and wireless interfaces’ IP addresses.  Now, I mount the NFS server, start copying a large file, then disconnect the Ethernet.  There is a brief pause as my OS adjusts to the change, then the progress resumes, albeit at a slower pace.  Sure enough, the data is now moving over wireless.  Why?  Well, let’s look at what happens when you disconnect an interface.  The computer notices that you’ve lost a link so it brings down the interface, killing all TCP connections (like your web download).  Any NFS data en route from the server to that interface gets lost along the way, but your computer notices it hasn’t arrived and re-requests it.  This request now goes out the wireless interface, and the NFS server sees the request coming from a different IP than before, but given that NFS is using stateless UDP it doesn’t really care so it sends the data along to the new IP.  And lo, your NFS transfer continues as if nothing happened (albeit slower due to the wireless’s lower speed).

And that is why I love NFS.