The Linker Is not a Magical Program - Revision history

Kevlin at 02:53, 19 July 2009

Kevlin — Sun, 19 Jul 2009 02:53:49 GMT

←Older revision		Revision as of 02:53, 19 July 2009
Line 18:		Line 18:
	So let's say the linker is saying <code>def is defined more than once</code>. Many programming languages, such as C, C++, and D, have both declarations and definitions. Declarations normally go into header files, like:		So let's say the linker is saying <code>def is defined more than once</code>. Many programming languages, such as C, C++, and D, have both declarations and definitions. Declarations normally go into header files, like:

-	~~<code>~~extern int iii;~~</code>~~	+	extern int iii;

	which generates an external reference to the symbol <code>iii</code>. A definition, on the other hand, actually sets aside storage for the symbol, usually appears in the implementation file, and looks like this:		which generates an external reference to the symbol <code>iii</code>. A definition, on the other hand, actually sets aside storage for the symbol, usually appears in the implementation file, and looks like this:

-	~~<code>~~int iii = 3;~~</code>~~	+	int iii = 3;

-	As in the film "Highlander~~", as far as definitions go~~, there can be only one ~~for each symbol~~. ~~Therefore~~, if a definition of <code>iii</code> appears in more than one implementation file~~, such as:~~	+	How many definitions can there be for each symbol? As in the film ''Highlander'', there can be only one. So, what if a definition of <code>iii</code> appears in more than one implementation file?

-	File a.c:	+	// File a.c
		+	int iii = 3;

-	~~<code>int~~ iii = 3;~~</code>~~	+	// File b.c
		+	double iii(int x) { return 3.7; }

-	~~File b~~.c:	+	The linker will complain about <code>iii</code> being multiply defined.

-	<code>~~double~~ iii~~(int x) { return 3.7; }~~</code>	+	Not only can there be only one, there must be one. If <code>iii</code> only appears as a declaration, but never a definition, the linker will complain about <code>iii</code> being an unresolved symbol.

-	~~then~~ the ~~linker will complain about <code>iii</code> being multiply defined~~.	+	To determine why an executable is the size it is, take a look at the map file that linkers optionally generate. A map file is nothing more than a list of all the symbols in the executable along with their addresses. This tells you what modules were linked in from the library, and the sizes of each module. Now you can see where the bloat is coming from. Often there will be library modules that you have no idea why were linked in. To figure it out, temporarily remove the suspicious module from the library, and relink. The undefined symbol error then generated will indicate who is referencing that module.
-		+
-	~~Not only~~ can there be ~~only one, there must be one~~. ~~If <code>iii</code> only appears as a declaration, but never a definition~~, the ~~linker will complain about <code>iii</code> being an unresolved~~ symbol.	+

-	~~To determine why an executable is the size~~ it is, ~~take a look at the map file that linkers~~	+	Although it is not always immediately obvious why you get a particular linker message, there is nothing magical about linkers. The mechanics are straightforward; it's the details you have to figure out in each case.
-	~~optionally generate. A map file~~ is nothing ~~more than a list of all the symbols in the executable~~	+
-	~~along with their addresses~~. ~~This tells you what modules were linked in from~~ the ~~library, and the~~	+
-	~~sizes of each module. Now you can see where the bloat is coming from. Often there will be library modules that~~ you have ~~no idea why were linked in. To~~ figure it out~~, temporarily remove the suspicious module from the library, and relink. The undefined symbol error then generated will indicate who is referencing that module~~.	+

	By [[Walter Bright]]		By [[Walter Bright]]

Kevlin: The Linker Is Not a Magical Program moved to The Linker Is not a Magical Program

Kevlin — Sun, 19 Jul 2009 02:33:10 GMT

The Linker Is Not a Magical Program moved to The Linker Is not a Magical Program

←Older revision

Revision as of 02:33, 19 July 2009

Walter Bright: Replaced second half.

Walter Bright — Sat, 21 Feb 2009 06:56:18 GMT

Replaced second half.

←Older revision		Revision as of 06:56, 21 February 2009
Line 16:		Line 16:
	A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it. No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.		A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it. No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.

-	So let's say the linker is saying <code>def is defined more than once</code>. ~~If it's a good~~	+	So let's say the linker is saying <code>def is defined more than once</code>. Many programming languages, such as C, C++, and D, have both declarations and definitions. Declarations normally go into header files, like:
-	~~linker~~, ~~it'll tell you which object file(s) the symbol appears in~~, and ~~so look at them~~ and	+
-	~~remove the duplicate~~. ~~If it doesn't tell you where the symbols are coming from~~, ~~the easy way~~	+
-	~~to figure it out is use grep~~:	+

-	<code>~~grep def *.obj~~</code>	+	<code>extern int iii;</code>

-	If, ~~once those files are identified~~, ~~it isn't clear from~~ the ~~corresponding source file where~~	+	which generates an external reference to the symbol <code>iii</code>. A definition, on the other hand, actually sets aside storage for the symbol, usually appears in the implementation file, and looks like this:
-	~~those symbols come from (can happen with templates)~~, ~~try examining~~ the ~~object files with a dumper~~	+
-	~~or disassembler.~~	+

-	~~If the linker is saying~~ <code>~~abc is an unresolved symbol~~</code>~~, it probably says which object file it was referenced from. If it doesn't, once again grep rides to the rescue:~~	+	<code>int iii = 3;</code>

-	<code>~~grep abc *.obj~~</code>	+	As in the film "Highlander", as far as definitions go, there can be only one for each symbol. Therefore, if a definition of <code>iii</code> appears in more than one implementation file, such as:

-	which will tell you which object file, and hence which source file, it comes from. Then it's up to you to figure out who is supposed to define it, and make sure that definition is supplied to the linker.	+	File a.c:

-	Often, though, these undefined symbols are defined in some system library. Given the long list of system libraries, it isn't immediately obvious which needs to be linked in to resolve the symbol. Needless to say, once again grep is the magic solution:	+	<code>int iii = 3;</code>

-	<code>grep abc \compiler\lib\*.~~lib</code>~~	+	File b.c:

-	~~or where ever the system libraries are stored~~. ~~The nice thing about using grep for this is it doesn't matter what format~~ the ~~object files or libraries are in~~. ~~I've never met~~ one ~~yet that didn't store symbol names~~ as ~~plaintext strings~~.	+	<code>double iii(int x) { return 3.7; }</code>
		+
		+	then the linker will complain about <code>iii</code> being multiply defined.
		+
		+	Not only can there be only one, there must be one. If <code>iii</code> only appears as a declaration, but never a definition, the linker will complain about <code>iii</code> being an unresolved symbol.

	To determine why an executable is the size it is, take a look at the map file that linkers		To determine why an executable is the size it is, take a look at the map file that linkers

Walter Bright at 00:31, 4 February 2009

Walter Bright — Wed, 04 Feb 2009 00:31:06 GMT

←Older revision		Revision as of 00:31, 4 February 2009
Line 8:		Line 8:
	Step 3 is, of course, the linking step. Why would I say such an outrageous thing? I've been doing tech support for decades, and I get the following questions again and again:		Step 3 is, of course, the linking step. Why would I say such an outrageous thing? I've been doing tech support for decades, and I get the following questions again and again:

-	# The linker says <code>abc</code> is an unresolved symbol.
	# The linker says <code>def</code> is defined more than once.		# The linker says <code>def</code> is defined more than once.
		+	# The linker says <code>abc</code> is an unresolved symbol.
	# Why is my executable so large?		# Why is my executable so large?

Line 15:		Line 15:

	A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it. No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.		A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it. No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.
		+
		+	So let's say the linker is saying <code>def is defined more than once</code>. If it's a good
		+	linker, it'll tell you which object file(s) the symbol appears in, and so look at them and
		+	remove the duplicate. If it doesn't tell you where the symbols are coming from, the easy way
		+	to figure it out is use grep:
		+
		+	<code>grep def *.obj</code>
		+
		+	If, once those files are identified, it isn't clear from the corresponding source file where
		+	those symbols come from (can happen with templates), try examining the object files with a dumper
		+	or disassembler.
		+
		+	If the linker is saying <code>abc is an unresolved symbol</code>, it probably says which object file it was referenced from. If it doesn't, once again grep rides to the rescue:
		+
		+	<code>grep abc *.obj</code>
		+
		+	which will tell you which object file, and hence which source file, it comes from. Then it's up to you to figure out who is supposed to define it, and make sure that definition is supplied to the linker.
		+
		+	Often, though, these undefined symbols are defined in some system library. Given the long list of system libraries, it isn't immediately obvious which needs to be linked in to resolve the symbol. Needless to say, once again grep is the magic solution:
		+
		+	<code>grep abc \compiler\lib\*.lib</code>
		+
		+	or where ever the system libraries are stored. The nice thing about using grep for this is it doesn't matter what format the object files or libraries are in. I've never met one yet that didn't store symbol names as plaintext strings.
		+
		+	To determine why an executable is the size it is, take a look at the map file that linkers
		+	optionally generate. A map file is nothing more than a list of all the symbols in the executable
		+	along with their addresses. This tells you what modules were linked in from the library, and the
		+	sizes of each module. Now you can see where the bloat is coming from. Often there will be library modules that you have no idea why were linked in. To figure it out, temporarily remove the suspicious module from the library, and relink. The undefined symbol error then generated will indicate who is referencing that module.

	By [[Walter Bright]]		By [[Walter Bright]]

Kevlin at 16:12, 3 February 2009

Kevlin — Tue, 03 Feb 2009 16:12:43 GMT

←Older revision		Revision as of 16:12, 3 February 2009
Line 1:		Line 1:
-	Depressingly often (happened to me again just before I wrote this), the view ~~a programmer has~~ of the process of going from source code to an executable in a compiled language is:	+	Depressingly often (happened to me again just before I wrote this), the view many programmers have of the process of going from source code to a statically linked executable in a compiled language is:

	# Edit source code		# Edit source code

Kevlin at 19:48, 22 November 2008

Kevlin — Sat, 22 Nov 2008 19:48:38 GMT

←Older revision		Revision as of 19:48, 22 November 2008
Line 8:		Line 8:
	Step 3 is, of course, the linking step. Why would I say such an outrageous thing? I've been doing tech support for decades, and I get the following questions again and again:		Step 3 is, of course, the linking step. Why would I say such an outrageous thing? I've been doing tech support for decades, and I get the following questions again and again:

-	# The linker says "abc" is an unresolved symbol.	+	# The linker says <code>abc</code> is an unresolved symbol.
-	# The linker says "def" is defined more than once.	+	# The linker says <code>def</code> is defined more than once.
	# Why is my executable so large?		# Why is my executable so large?

Line 15:		Line 15:

	A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it. No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.		A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it. No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.
		+
		+	By [[Walter Bright]]
		+
		+	This work is licensed under a [http://creativecommons.org/licenses/by/3.0/us/ Creative Commons Attribution 3]
		+
		+
		+
		+	Back to [[97 Things Every Programmer Should Know]] home page

Kevlin at 17:46, 22 November 2008

Kevlin — Sat, 22 Nov 2008 17:46:53 GMT

←Older revision		Revision as of 17:46, 22 November 2008
Line 1:		Line 1:
-	Depressingly often (~~just~~ happened to me again ~~today~~), the view a programmer has of the process of going from source code to an executable is:	+	Depressingly often (happened to me again just before I wrote this), the view a programmer has of the process of going from source code to an executable in a compiled language is:

-	1. Edit source code	+	# Edit source code
-	2. Compile source code into object files	+	# Compile source code into object files
-	3. Something magical happens	+	# Something magical happens
-	4. Run executable	+	# Run executable

	Step 3 is, of course, the linking step. Why would I say such an outrageous thing? I've been doing tech support for decades, and I get the following questions again and again:		Step 3 is, of course, the linking step. Why would I say such an outrageous thing? I've been doing tech support for decades, and I get the following questions again and again:

-	1. The linker says "abc" is an unresolved symbol.	+	# The linker says "abc" is an unresolved symbol.
-	2. The linker says "def" is defined more than once.	+	# The linker says "def" is defined more than once.
-	3. Why is my executable so large?	+	# Why is my executable so large?

-	Followed by "~~what~~ do I do now" usually with the ~~words~~ "seems to" and "somehow" mixed in, and an aura of utter bafflement. It's the "seems to" and "somehow" that indicate that the linking process is viewed as a magical process, presumably understandable by only wizards and warlocks. The process of compiling does not elicit these kinds of phrases, implying that programmers generally understand how compilers work, or at least what they do.	+	Followed by "What do I do now?" usually with the phrases "seems to" and "somehow" mixed in, and an aura of utter bafflement. It's the "seems to" and "somehow" that indicate that the linking process is viewed as a magical process, presumably understandable only by wizards and warlocks. The process of compiling does not elicit these kinds of phrases, implying that programmers generally understand how compilers work, or at least what they do.

-	A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it! No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.	+	A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it. No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.

Walter Bright at 08:17, 21 November 2008

Walter Bright — Fri, 21 Nov 2008 08:17:11 GMT

←Older revision		Revision as of 08:17, 21 November 2008
Line 1:		Line 1:
-	Depressingly often, the view a programmer has of the process of going from source code to an executable is:	+	Depressingly often (just happened to me again today), the view a programmer has of the process of going from source code to an executable is:

	1. Edit source code		1. Edit source code
Line 12:		Line 12:
	3. Why is my executable so large?		3. Why is my executable so large?

-	Followed by "what do I do now" usually with the words "seems to" and "somehow" mixed in. It's the "seems to" and "somehow" that indicate that the linking process is viewed as a magical process, presumably understandable by only wizards and warlocks. The process of compiling does not elicit these kinds of phrases, implying that programmers generally understand how compilers work, or at least what they do.	+	Followed by "what do I do now" usually with the words "seems to" and "somehow" mixed in, and an aura of utter bafflement. It's the "seems to" and "somehow" that indicate that the linking process is viewed as a magical process, presumably understandable by only wizards and warlocks. The process of compiling does not elicit these kinds of phrases, implying that programmers generally understand how compilers work, or at least what they do.

	A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it! No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.		A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it! No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.

Walter Bright: New page: Depressingly often, the view a programmer has of the process of going from source code to an executable is: 1. Edit source code 2. Compile source code into object files 3. Something magic...

Walter Bright — Fri, 21 Nov 2008 08:12:05 GMT

New page: Depressingly often, the view a programmer has of the process of going from source code to an executable is: 1. Edit source code 2. Compile source code into object files 3. Something magic...

New page

Depressingly often, the view a programmer has of the process of going from source code to an executable is:

1. Edit source code
2. Compile source code into object files
3. Something magical happens
4. Run executable

Step 3 is, of course, the linking step. Why would I say such an outrageous thing? I've been doing tech support for decades, and I get the following questions again and again:

1. The linker says "abc" is an unresolved symbol.
2. The linker says "def" is defined more than once.
3. Why is my executable so large?

Followed by "what do I do now" usually with the words "seems to" and "somehow" mixed in. It's the "seems to" and "somehow" that indicate that the linking process is viewed as a magical process, presumably understandable by only wizards and warlocks. The process of compiling does not elicit these kinds of phrases, implying that programmers generally understand how compilers work, or at least what they do.

A linker is a very stupid, pedestrian, straightforward program. All it does is concatenate together the code and data sections of the object files, connect the the references to symbols with their definitions, pull unresolved symbols out of the library, and write out an executable. That's it! No spells! No magic! The tedium in writing a linker is usually all about decoding and generating the usually ridiculously over-complicated file formats, but that doesn't change the essential nature of a linker.